2024-03-26T02:08:10.650 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T02:08:10.650 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368 --owner scheduled_adking@teuthology --verbose --description orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} --name adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi -- /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/orig.config.yaml 2024-03-26T02:08:10.698 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T02:08:10.786 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368 branch: wip-adk-testing-2024-03-25-1628 description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} email: adking@redhat.com first_in_suite: false job_id: '7622368' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-03-25-1628 ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN - mons down, quorum - is down \(out of quorum\) - PG_DEGRADED - PG_AVAILABILITY - OSD_DOWN sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 workunit: branch: wip-adk-testing-2024-03-25-1628 sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 9192 sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-03-25-1628 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 targets: smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9Y+jJxG3SK/CYTEVLI5sHtamX0DnyUoB9Jg8lZWQEEnoq0QFqWuPnWjklx0lJurNmOgobtycmv3VaDE22/M9o= smithi172.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPopve8QBzZsUc6kyh4uSWmx3Pvp6NWnL9MgU1L9kbkSmqmqf4tyO4SXov0zAASIkbM+YdljgWvlfNwgq0AGbSc= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 180 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch ps - ceph versions - ceph -s - ceph orch ls - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 2' - ceph mgr fail - sleep 180 - ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1 - ceph orch ps --refresh - sleep 180 - ceph orch ps - ceph versions - ceph health detail - ceph -s - ceph mgr fail - sleep 180 - ceph orch ps - ceph versions - ceph -s - ceph health detail - ceph versions | jq -e '.mgr | length == 1' - ceph mgr fail - sleep 180 - ceph orch ps - ceph orch ls - ceph versions - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 2' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '{print $2}') - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.mon | length == 1' - ceph versions | jq -e '.mon | keys' | grep $sha1 - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 5' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 7' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1 - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 8' - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.osd | length == 1' - ceph versions | jq -e '.osd | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions | jq -e '.rgw | length == 1' - ceph versions | jq -e '.rgw | keys' | grep $sha1 - ceph orch upgrade status - ceph health detail - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_01:50:53 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T02:08:10.786 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa; will attempt to use it 2024-03-26T02:08:10.787 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa/tasks 2024-03-26T02:08:10.787 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T02:08:10.789 INFO:teuthology.task.internal:Checking packages... 2024-03-26T02:08:10.813 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003' 2024-03-26T02:08:10.813 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T02:08:10.813 INFO:teuthology.packaging:ref: None 2024-03-26T02:08:10.813 INFO:teuthology.packaging:tag: None 2024-03-26T02:08:10.813 INFO:teuthology.packaging:branch: wip-adk-testing-2024-03-25-1628 2024-03-26T02:08:10.814 INFO:teuthology.packaging:sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:08:10.814 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-03-25-1628 2024-03-26T02:08:10.954 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2475.ge42b8ea3 2024-03-26T02:08:10.955 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T02:08:10.962 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T02:08:10.962 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T02:08:11.038 INFO:teuthology.task.internal:Saving configuration 2024-03-26T02:08:11.056 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T02:08:11.062 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:08:11.088 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 02:03:01.602044', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9Y+jJxG3SK/CYTEVLI5sHtamX0DnyUoB9Jg8lZWQEEnoq0QFqWuPnWjklx0lJurNmOgobtycmv3VaDE22/M9o='} 2024-03-26T02:08:11.111 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 02:03:01.603484', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPopve8QBzZsUc6kyh4uSWmx3Pvp6NWnL9MgU1L9kbkSmqmqf4tyO4SXov0zAASIkbM+YdljgWvlfNwgq0AGbSc='} 2024-03-26T02:08:11.111 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T02:08:11.117 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T02:08:11.117 INFO:teuthology.task.internal:roles: ubuntu@smithi172.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-03-26T02:08:11.117 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T02:08:11.194 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f127f9c6160>, signals=[15]) 2024-03-26T02:08:11.194 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T02:08:11.200 INFO:teuthology.task.internal:Opening connections... 2024-03-26T02:08:11.201 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-03-26T02:08:11.203 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:08:11.294 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi172.front.sepia.ceph.com 2024-03-26T02:08:11.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:08:11.384 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T02:08:11.390 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-03-26T02:08:11.408 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-03-26T02:08:11.409 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-03-26T02:08:11.466 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-03-26T02:08:11.466 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-03-26T02:08:11.466 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-03-26T02:08:11.466 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T02:08:11.467 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T02:08:11.468 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-03-26T02:08:11.492 DEBUG:teuthology.orchestra.run.smithi172:> uname -m 2024-03-26T02:08:11.510 INFO:teuthology.orchestra.run.smithi172.stdout:x86_64 2024-03-26T02:08:11.510 DEBUG:teuthology.orchestra.run.smithi172:> cat /etc/os-release 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:NAME="CentOS Stream" 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION="9" 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:ID="centos" 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:ID_LIKE="rhel fedora" 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:VERSION_ID="9" 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:PLATFORM_ID="platform:el9" 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:ANSI_COLOR="0;31" 2024-03-26T02:08:11.568 INFO:teuthology.orchestra.run.smithi172.stdout:LOGO="fedora-logo-icon" 2024-03-26T02:08:11.569 INFO:teuthology.orchestra.run.smithi172.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T02:08:11.569 INFO:teuthology.orchestra.run.smithi172.stdout:HOME_URL="https://centos.org/" 2024-03-26T02:08:11.569 INFO:teuthology.orchestra.run.smithi172.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T02:08:11.569 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T02:08:11.569 INFO:teuthology.orchestra.run.smithi172.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T02:08:11.570 INFO:teuthology.lock.ops:Updating smithi172.front.sepia.ceph.com on lock server 2024-03-26T02:08:11.591 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T02:08:11.600 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T02:08:11.606 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T02:08:11.607 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-03-26T02:08:11.609 DEBUG:teuthology.orchestra.run.smithi172:> test '!' -e /home/ubuntu/cephtest 2024-03-26T02:08:11.625 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T02:08:11.631 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T02:08:11.631 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-03-26T02:08:11.666 DEBUG:teuthology.orchestra.run.smithi172:> test -z $(ls -A /var/lib/ceph) 2024-03-26T02:08:11.760 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T02:08:11.806 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T02:08:11.821 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T02:08:11.821 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T02:08:11.821 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T02:08:11.822 DEBUG:teuthology.orchestra.run.smithi172:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T02:08:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:08:11.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:08:11.840 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-26T02:08:11.840 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-03-26T02:08:11.896 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-361.el9.x86_64 2024-03-26T02:08:11.897 INFO:teuthology.task.kernel:Running kernel on smithi172: 5.14.0-361.el9.x86_64 2024-03-26T02:08:11.897 DEBUG:teuthology.orchestra.run.smithi172:> sudo yum install -y kernel 2024-03-26T02:08:11.898 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-361.el9.x86_64 2024-03-26T02:08:11.898 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.14.0-361.el9.x86_64 2024-03-26T02:08:11.898 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2024-03-26T02:08:14.595 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - AppStream 6.1 MB/s | 6.7 MB 00:01 2024-03-26T02:08:14.950 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - AppStream 4.6 MB/s | 6.7 MB 00:01 2024-03-26T02:08:16.220 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-26T02:08:16.600 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-26T02:08:17.044 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-26T02:08:20.921 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:03 2024-03-26T02:08:22.516 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - AppStream 34 MB/s | 19 MB 00:00 2024-03-26T02:08:26.367 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-26T02:08:28.127 INFO:teuthology.orchestra.run.smithi172.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-03-26T02:08:31.505 INFO:teuthology.orchestra.run.smithi172.stdout:Extra Packages for Enterprise Linux 6.3 MB/s | 21 MB 00:03 2024-03-26T02:08:35.309 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:17 2024-03-26T02:08:36.302 INFO:teuthology.orchestra.run.smithi172.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-26T02:08:38.467 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T02:08:38.467 INFO:teuthology.orchestra.run.smithi172.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T02:08:38.567 INFO:teuthology.orchestra.run.smithi172.stdout:Dependencies resolved. 2024-03-26T02:08:38.574 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout: Package Architecture Version Repository Size 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout:Installing: 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout:Installing dependencies: 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction Summary 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout:================================================================================ 2024-03-26T02:08:38.575 INFO:teuthology.orchestra.run.smithi172.stdout:Install 4 Packages 2024-03-26T02:08:38.576 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:08:38.576 INFO:teuthology.orchestra.run.smithi172.stdout:Total download size: 76 M 2024-03-26T02:08:38.576 INFO:teuthology.orchestra.run.smithi172.stdout:Installed size: 123 M 2024-03-26T02:08:38.576 INFO:teuthology.orchestra.run.smithi172.stdout:Downloading Packages: 2024-03-26T02:08:38.661 INFO:teuthology.orchestra.run.smithi172.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 1.3 MB/s | 66 kB 00:00 2024-03-26T02:08:39.253 INFO:teuthology.orchestra.run.smithi172.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-03-26T02:08:39.651 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.1 MB 00:00 2024-03-26T02:08:39.853 INFO:teuthology.orchestra.run.smithi172.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 27 MB/s | 33 MB 00:01 2024-03-26T02:08:40.479 INFO:teuthology.orchestra.run.smithi172.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 15 MB/s | 27 MB 00:01 2024-03-26T02:08:40.479 INFO:teuthology.orchestra.run.smithi172.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:08:40.479 INFO:teuthology.orchestra.run.smithi172.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-26T02:08:41.143 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction check 2024-03-26T02:08:41.269 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-03-26T02:08:41.502 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction check succeeded. 2024-03-26T02:08:41.502 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction test 2024-03-26T02:08:42.406 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-26T02:08:42.515 INFO:teuthology.orchestra.run.smithi172.stdout:Transaction test succeeded. 2024-03-26T02:08:42.515 INFO:teuthology.orchestra.run.smithi172.stdout:Running transaction 2024-03-26T02:08:44.971 INFO:teuthology.orchestra.run.smithi172.stdout: Preparing : 1/1 2024-03-26T02:08:45.582 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T02:08:45.663 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:08:47.253 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-03-26T02:08:48.408 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:08:48.542 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:08:49.377 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T02:08:49.377 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T02:08:49.466 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-03-26T02:08:49.478 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T02:08:49.478 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-03-26T02:08:49.478 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T02:08:49.478 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-03-26T02:08:49.478 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T02:08:49.478 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-03-26T02:08:49.478 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T02:08:49.479 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T02:08:49.479 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T02:08:49.479 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:08:49.479 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-03-26T02:08:49.479 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-03-26T02:08:49.479 INFO:teuthology.orchestra.run.smithi164.stdout:Install 4 Packages 2024-03-26T02:08:49.479 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:08:49.480 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 76 M 2024-03-26T02:08:49.480 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 123 M 2024-03-26T02:08:49.480 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-03-26T02:08:49.584 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:08:49.584 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:08:49.584 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:08:49.727 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:08:49.737 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:08:49.760 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:08:49.796 INFO:teuthology.orchestra.run.smithi164.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 238 kB/s | 66 kB 00:00 2024-03-26T02:08:49.829 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:08:49.864 INFO:teuthology.orchestra.run.smithi164.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T02:08:50.380 INFO:teuthology.orchestra.run.smithi164.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-26T02:08:50.955 INFO:teuthology.orchestra.run.smithi164.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 23 MB/s | 33 MB 00:01 2024-03-26T02:08:51.490 INFO:teuthology.orchestra.run.smithi164.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-26T02:08:51.491 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-03-26T02:08:51.491 INFO:teuthology.orchestra.run.smithi164.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-26T02:08:52.159 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-03-26T02:08:52.539 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-03-26T02:08:52.539 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-03-26T02:08:53.230 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:08:53.267 INFO:teuthology.orchestra.run.smithi172.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:08:53.548 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-03-26T02:08:53.548 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-03-26T02:08:56.016 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-03-26T02:08:56.632 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T02:08:56.722 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:08:57.977 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:08:59.491 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:08:59.625 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:09:04.366 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:09:04.491 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:09.225 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:27.639 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:27.640 INFO:teuthology.orchestra.run.smithi172.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T02:09:27.640 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:09:27.684 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T02:09:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:09:28.034 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:09:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:09:28.444 INFO:teuthology.orchestra.run.smithi172.stdout:Installed: 2024-03-26T02:09:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T02:09:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T02:09:28.444 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T02:09:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T02:09:28.445 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:09:28.445 INFO:teuthology.orchestra.run.smithi172.stdout:Complete! 2024-03-26T02:09:28.766 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T02:09:28.863 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T02:09:28.864 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T02:09:28.864 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T02:09:28.864 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T02:09:28.864 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T02:09:28.864 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T02:09:28.864 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi172.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T02:09:28.864 DEBUG:teuthology.orchestra.run.smithi172:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T02:09:28.944 INFO:teuthology.orchestra.run.smithi172.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T02:09:28.945 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T02:09:28.945 DEBUG:teuthology.orchestra.run.smithi172:> sudo rpm -qi grub2-tools 2024-03-26T02:09:28.982 INFO:teuthology.orchestra.run.smithi172.stdout:Name : grub2-tools 2024-03-26T02:09:28.982 INFO:teuthology.orchestra.run.smithi172.stdout:Epoch : 1 2024-03-26T02:09:28.982 INFO:teuthology.orchestra.run.smithi172.stdout:Version : 2.06 2024-03-26T02:09:28.982 INFO:teuthology.orchestra.run.smithi172.stdout:Release : 68.el9 2024-03-26T02:09:28.982 INFO:teuthology.orchestra.run.smithi172.stdout:Architecture: x86_64 2024-03-26T02:09:28.982 INFO:teuthology.orchestra.run.smithi172.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Group : Unspecified 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Size : 8264254 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:License : GPLv3+ 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Packager : builder@centos.org 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Vendor : CentOS 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Summary : Support tools for GRUB. 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:Description : 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:09:28.983 INFO:teuthology.orchestra.run.smithi172.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T02:09:28.984 INFO:teuthology.orchestra.run.smithi172.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T02:09:28.984 INFO:teuthology.orchestra.run.smithi172.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T02:09:28.984 INFO:teuthology.orchestra.run.smithi172.stdout:hardware devices. 2024-03-26T02:09:28.984 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:09:28.984 INFO:teuthology.orchestra.run.smithi172.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T02:09:28.985 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T02:09:28.985 INFO:teuthology.task.kernel:Updating grub on smithi172 to boot 5.14.0-432.el9.x86_64 2024-03-26T02:09:28.985 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T02:09:29.570 INFO:teuthology.orchestra.run.smithi172.stderr:Generating grub configuration file ... 2024-03-26T02:09:30.899 INFO:teuthology.orchestra.run.smithi172.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T02:09:30.936 INFO:teuthology.orchestra.run.smithi172.stderr:done 2024-03-26T02:09:30.939 DEBUG:teuthology.orchestra.run.smithi172:> mktemp 2024-03-26T02:09:30.954 INFO:teuthology.orchestra.run.smithi172.stdout:/tmp/tmp.0Ytj3yacpJ 2024-03-26T02:09:30.954 DEBUG:teuthology.orchestra.run.smithi172:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0Ytj3yacpJ 2024-03-26T02:09:31.022 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0666 /tmp/tmp.0Ytj3yacpJ 2024-03-26T02:09:31.165 DEBUG:teuthology.orchestra.remote:smithi172:/tmp/tmp.0Ytj3yacpJ is 6KB 2024-03-26T02:09:31.214 DEBUG:teuthology.orchestra.run.smithi172:> rm -fr /tmp/tmp.0Ytj3yacpJ 2024-03-26T02:09:31.228 DEBUG:teuthology.orchestra.run.smithi172:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T02:09:31.293 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T02:09:31.294 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T02:09:31.294 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T02:09:31.294 INFO:teuthology.orchestra.run.smithi172.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T02:09:31.295 DEBUG:teuthology.orchestra.run.smithi172:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T02:09:31.448 DEBUG:teuthology.orchestra.run.smithi172:> sudo shutdown -r now 2024-03-26T02:09:38.741 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:38.742 INFO:teuthology.orchestra.run.smithi164.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T02:09:38.742 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:09:38.787 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:39.104 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:39.105 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T02:09:39.105 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T02:09:39.105 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:09:39.516 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-03-26T02:09:39.877 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T02:09:39.974 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T02:09:39.974 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T02:09:39.974 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T02:09:39.974 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T02:09:39.975 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T02:09:39.975 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T02:09:39.975 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T02:09:39.975 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T02:09:40.055 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T02:09:40.055 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T02:09:40.056 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2024-03-26T02:09:40.099 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2024-03-26T02:09:40.099 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2024-03-26T02:09:40.099 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.06 2024-03-26T02:09:40.099 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 68.el9 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Group : Unspecified 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8264254 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : builder@centos.org 2024-03-26T02:09:40.100 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:09:40.101 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T02:09:40.102 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T02:09:40.102 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 5.14.0-432.el9.x86_64 2024-03-26T02:09:40.102 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T02:09:40.712 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-03-26T02:09:42.093 INFO:teuthology.orchestra.run.smithi164.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T02:09:42.124 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-03-26T02:09:42.126 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-03-26T02:09:42.141 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.lnbrHJqXA9 2024-03-26T02:09:42.141 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lnbrHJqXA9 2024-03-26T02:09:42.210 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.lnbrHJqXA9 2024-03-26T02:09:42.348 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.lnbrHJqXA9 is 6KB 2024-03-26T02:09:42.396 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.lnbrHJqXA9 2024-03-26T02:09:42.410 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T02:09:42.477 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T02:09:42.477 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T02:09:42.478 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T02:09:42.478 INFO:teuthology.orchestra.run.smithi164.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T02:09:42.479 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T02:09:42.630 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-03-26T02:10:01.463 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T02:10:01.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-26T02:10:01.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:10:12.639 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T02:10:12.639 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T02:10:12.640 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:10:19.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-26T02:10:28.814 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-26T02:10:28.815 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:10:31.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-26T02:10:31.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.172 2024-03-26T02:10:40.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T02:10:40.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:10:43.134 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-03-26T02:10:43.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-26T02:10:43.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:10:55.146 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T02:10:55.147 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:11:43.923 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T02:11:55.158 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T02:11:58.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-26T02:11:58.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:11:59.264 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-26T02:11:59.669 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-26T02:11:59.669 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T02:11:59.670 DEBUG:teuthology.orchestra.run.smithi172:> uname -r 2024-03-26T02:11:59.685 INFO:teuthology.orchestra.run.smithi172.stdout:5.14.0-432.el9.x86_64 2024-03-26T02:11:59.686 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T02:11:59.686 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T02:11:59.686 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T02:12:00.687 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T02:12:00.687 DEBUG:teuthology.orchestra.run.smithi172:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T02:12:00.780 INFO:teuthology.orchestra.run.smithi172.stdout:ttyS1 2024-03-26T02:12:00.817 DEBUG:teuthology.parallel:result is None 2024-03-26T02:12:10.159 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T02:12:10.160 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:12:10.577 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-26T02:12:10.980 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T02:12:10.980 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T02:12:10.980 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-03-26T02:12:10.996 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-432.el9.x86_64 2024-03-26T02:12:10.996 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T02:12:10.996 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T02:12:10.996 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T02:12:11.997 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T02:12:11.998 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T02:12:12.070 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-03-26T02:12:12.104 DEBUG:teuthology.parallel:result is None 2024-03-26T02:12:12.104 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T02:12:12.111 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T02:12:12.112 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T02:12:12.114 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T02:12:12.132 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T02:12:12.138 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T02:12:12.174 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T02:12:12.174 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T02:12:12.177 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T02:12:12.270 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T02:12:12.277 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T02:12:12.277 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T02:12:12.280 DEBUG:teuthology.orchestra.run.smithi172:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T02:12:12.320 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:12:12.326 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:12:12.334 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:12:12.340 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T02:12:12.342 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T02:12:12.350 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T02:12:12.350 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T02:12:12.352 DEBUG:teuthology.orchestra.run.smithi172:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T02:12:12.405 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T02:12:12.419 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T02:12:12.419 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T02:12:12.422 DEBUG:teuthology.orchestra.run.smithi172:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T02:12:12.461 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:12:12.520 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:12:12.573 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:12:12.656 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:12:12.699 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:12:12.699 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T02:12:12.767 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:12:12.811 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T02:12:12.864 DEBUG:teuthology.orchestra.run.smithi172:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:12:12.948 DEBUG:teuthology.orchestra.run.smithi172:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T02:12:12.994 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-26T02:12:12.994 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T02:12:13.063 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-03-26T02:12:13.066 DEBUG:teuthology.orchestra.run.smithi172:> sudo service rsyslog restart 2024-03-26T02:12:13.100 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:12:13.148 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:12:13.591 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T02:12:13.598 INFO:teuthology.task.internal:Starting timer... 2024-03-26T02:12:13.599 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T02:12:13.609 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T02:12:13.619 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-03-26T02:12:13.681 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-03-26T02:12:13.683 DEBUG:teuthology.orchestra.run.smithi172:> sudo service auditd rotate 2024-03-26T02:12:13.730 INFO:teuthology.orchestra.run.smithi172.stdout:Rotating logs: 2024-03-26T02:12:13.732 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T02:12:13.732 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-03-26T02:12:13.749 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-03-26T02:12:13.749 DEBUG:teuthology.orchestra.run.smithi172:> /usr/sbin/getenforce 2024-03-26T02:12:13.781 INFO:teuthology.orchestra.run.smithi172.stdout:Permissive 2024-03-26T02:12:13.782 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi164.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi172.front.sepia.ceph.com': 'permissive'} 2024-03-26T02:12:13.782 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-26T02:12:13.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:13.819 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-26T02:12:13.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:12:13.851 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T02:12:13.851 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-03-26T02:12:13.893 DEBUG:teuthology.orchestra.run.smithi172:> sudo /usr/sbin/setenforce permissive 2024-03-26T02:12:13.930 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T02:12:13.940 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T02:12:13.957 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T02:12:13.959 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi164.front.sepia.ceph.com,smithi172.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T02:16:40.837 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi164.front.sepia.ceph.com'), Remote(name='ubuntu@smithi172.front.sepia.ceph.com')] 2024-03-26T02:16:40.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T02:16:40.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:16:40.932 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-03-26T02:16:41.009 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-03-26T02:16:41.010 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-26T02:16:41.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi172.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T02:16:41.098 DEBUG:teuthology.orchestra.run.smithi172:> true 2024-03-26T02:16:41.182 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi172.front.sepia.ceph.com' 2024-03-26T02:16:41.182 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T02:16:41.202 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T02:16:41.202 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T02:16:41.203 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:16:41.205 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T02:16:41.205 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:16:41.233 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T02:16:41.250 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T02:16:41.253 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T02:16:41.270 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T02:16:41.279 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-03-26T02:16:41.292 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-03-26T02:16:41.299 INFO:teuthology.orchestra.run.smithi172.stderr:sudo: ntpd: command not found 2024-03-26T02:16:41.309 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T02:16:41.313 INFO:teuthology.orchestra.run.smithi172.stdout:506 Cannot talk to daemon 2024-03-26T02:16:41.325 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T02:16:41.329 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T02:16:41.345 INFO:teuthology.orchestra.run.smithi172.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T02:16:41.383 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:16:41.387 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:16:41.387 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-26T02:16:41.387 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:16:41.387 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:16:41.387 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:16:41.387 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:16:41.398 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:16:41.402 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:16:41.402 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-26T02:16:41.402 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:16:41.402 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:16:41.402 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:16:41.402 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:16:41.403 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T02:16:41.515 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'mons down, quorum', 'is down \\(out of quorum\\)', 'PG_DEGRADED', 'PG_AVAILABILITY', 'OSD_DOWN'], 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'} 2024-03-26T02:16:41.515 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-03-26T02:16:41.516 INFO:tasks.cephadm:Cluster fsid is e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:16:41.516 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T02:16:41.516 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.164', 'mon.c': '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]', 'mon.b': '172.21.15.172'} 2024-03-26T02:16:41.516 INFO:tasks.cephadm:First mon is mon.a on smithi164 2024-03-26T02:16:41.516 INFO:tasks.cephadm:First mgr is y 2024-03-26T02:16:41.516 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T02:16:41.517 DEBUG:teuthology.orchestra.run.smithi164:> sudo hostname $(hostname -s) 2024-03-26T02:16:41.545 DEBUG:teuthology.orchestra.run.smithi172:> sudo hostname $(hostname -s) 2024-03-26T02:16:41.573 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-03-26T02:16:41.573 DEBUG:teuthology.orchestra.run.smithi164:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T02:16:42.002 INFO:teuthology.orchestra.run.smithi164.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 26 02:16 /home/ubuntu/cephtest/cephadm 2024-03-26T02:16:42.003 DEBUG:teuthology.orchestra.run.smithi172:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T02:16:42.384 INFO:teuthology.orchestra.run.smithi172.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Mar 26 02:16 /home/ubuntu/cephtest/cephadm 2024-03-26T02:16:42.384 DEBUG:teuthology.orchestra.run.smithi164:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T02:16:42.400 DEBUG:teuthology.orchestra.run.smithi172:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T02:16:42.426 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-03-26T02:16:42.426 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-26T02:16:42.442 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-03-26T02:16:43.347 INFO:teuthology.orchestra.run.smithi172.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T02:16:43.386 INFO:teuthology.orchestra.run.smithi164.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T02:17:04.573 INFO:teuthology.orchestra.run.smithi172.stdout:{ 2024-03-26T02:17:04.574 INFO:teuthology.orchestra.run.smithi172.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-26T02:17:04.574 INFO:teuthology.orchestra.run.smithi172.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-26T02:17:04.574 INFO:teuthology.orchestra.run.smithi172.stdout: "repo_digests": [ 2024-03-26T02:17:04.574 INFO:teuthology.orchestra.run.smithi172.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-26T02:17:04.574 INFO:teuthology.orchestra.run.smithi172.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-26T02:17:04.574 INFO:teuthology.orchestra.run.smithi172.stdout: ] 2024-03-26T02:17:04.574 INFO:teuthology.orchestra.run.smithi172.stdout:} 2024-03-26T02:17:05.021 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:17:05.022 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-03-26T02:17:05.022 INFO:teuthology.orchestra.run.smithi164.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-03-26T02:17:05.022 INFO:teuthology.orchestra.run.smithi164.stdout: "repo_digests": [ 2024-03-26T02:17:05.022 INFO:teuthology.orchestra.run.smithi164.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-03-26T02:17:05.022 INFO:teuthology.orchestra.run.smithi164.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-03-26T02:17:05.022 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-03-26T02:17:05.022 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:17:05.040 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph 2024-03-26T02:17:05.093 DEBUG:teuthology.orchestra.run.smithi172:> sudo mkdir -p /etc/ceph 2024-03-26T02:17:05.125 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /etc/ceph 2024-03-26T02:17:05.151 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 777 /etc/ceph 2024-03-26T02:17:05.191 INFO:tasks.cephadm:Writing seed config... 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T02:17:05.193 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T02:17:05.194 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:17:05.194 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T02:17:05.209 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e2654d3c-eb16-11ee-95d0-87774f69a715 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-03-26T02:17:05.209 DEBUG:teuthology.orchestra.run.smithi164:mon.a> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T02:17:05.251 DEBUG:teuthology.orchestra.run.smithi164:mgr.y> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T02:17:05.294 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T02:17:05.294 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.164 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:17:05.488 INFO:teuthology.orchestra.run.smithi164.stderr:-------------------------------------------------------------------------------- 2024-03-26T02:17:05.488 INFO:teuthology.orchestra.run.smithi164.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'e2654d3c-eb16-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.164', '--skip-admin-label'] 2024-03-26T02:17:05.511 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: 4.9.4-dev 2024-03-26T02:17:05.513 INFO:teuthology.orchestra.run.smithi164.stderr:Verifying podman|docker is present... 2024-03-26T02:17:05.534 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: 4.9.4-dev 2024-03-26T02:17:05.536 INFO:teuthology.orchestra.run.smithi164.stderr:Verifying lvm2 is present... 2024-03-26T02:17:05.536 INFO:teuthology.orchestra.run.smithi164.stderr:Verifying time synchronization is in place... 2024-03-26T02:17:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T02:17:05.549 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: inactive 2024-03-26T02:17:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: enabled 2024-03-26T02:17:05.561 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: active 2024-03-26T02:17:05.562 INFO:teuthology.orchestra.run.smithi164.stderr:Unit chronyd.service is enabled and running 2024-03-26T02:17:05.562 INFO:teuthology.orchestra.run.smithi164.stderr:Repeating the final host check... 2024-03-26T02:17:05.584 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: 4.9.4-dev 2024-03-26T02:17:05.586 INFO:teuthology.orchestra.run.smithi164.stderr:podman (/bin/podman) version 4.9.4 is present 2024-03-26T02:17:05.586 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl is present 2024-03-26T02:17:05.586 INFO:teuthology.orchestra.run.smithi164.stderr:lvcreate is present 2024-03-26T02:17:05.592 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T02:17:05.598 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: inactive 2024-03-26T02:17:05.604 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: enabled 2024-03-26T02:17:05.610 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: active 2024-03-26T02:17:05.611 INFO:teuthology.orchestra.run.smithi164.stderr:Unit chronyd.service is enabled and running 2024-03-26T02:17:05.611 INFO:teuthology.orchestra.run.smithi164.stderr:Host looks OK 2024-03-26T02:17:05.611 INFO:teuthology.orchestra.run.smithi164.stderr:Cluster fsid: e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:05.611 INFO:teuthology.orchestra.run.smithi164.stderr:Acquiring lock 140607821668256 on /run/cephadm/e2654d3c-eb16-11ee-95d0-87774f69a715.lock 2024-03-26T02:17:05.611 INFO:teuthology.orchestra.run.smithi164.stderr:Lock 140607821668256 acquired on /run/cephadm/e2654d3c-eb16-11ee-95d0-87774f69a715.lock 2024-03-26T02:17:05.612 INFO:teuthology.orchestra.run.smithi164.stderr:Verifying IP 172.21.15.164 port 3300 ... 2024-03-26T02:17:05.612 INFO:teuthology.orchestra.run.smithi164.stderr:Verifying IP 172.21.15.164 port 6789 ... 2024-03-26T02:17:05.612 INFO:teuthology.orchestra.run.smithi164.stderr:Base mon IP is 172.21.15.164, final addrv is [v2:172.21.15.164:3300,v1:172.21.15.164:6789] 2024-03-26T02:17:05.614 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.164 metric 100 2024-03-26T02:17:05.614 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.164 metric 100 2024-03-26T02:17:05.617 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T02:17:05.617 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T02:17:05.617 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1775sec hoplimit 64 pref medium 2024-03-26T02:17:05.620 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T02:17:05.620 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-03-26T02:17:05.620 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-26T02:17:05.620 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T02:17:05.620 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:73f3/64 scope link 2024-03-26T02:17:05.620 INFO:teuthology.orchestra.run.smithi164.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-03-26T02:17:05.621 INFO:teuthology.orchestra.run.smithi164.stderr:Mon IP `172.21.15.164` is in CIDR network `172.21.0.0/20` 2024-03-26T02:17:05.621 INFO:teuthology.orchestra.run.smithi164.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T02:17:05.621 INFO:teuthology.orchestra.run.smithi164.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-03-26T02:17:05.721 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-03-26T02:17:06.285 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Getting image source signatures 2024-03-26T02:17:06.285 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-03-26T02:17:06.285 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-03-26T02:17:06.285 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-03-26T02:17:06.285 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-03-26T02:17:06.285 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-03-26T02:17:06.285 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-26T02:17:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: Writing manifest to image destination 2024-03-26T02:17:06.611 INFO:teuthology.orchestra.run.smithi164.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-03-26T02:17:07.034 INFO:teuthology.orchestra.run.smithi164.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-26T02:17:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-03-26T02:17:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:Extracting ceph user uid/gid from container image... 2024-03-26T02:17:07.877 INFO:teuthology.orchestra.run.smithi164.stderr:stat: 167 167 2024-03-26T02:17:08.449 INFO:teuthology.orchestra.run.smithi164.stderr:Creating initial keys... 2024-03-26T02:17:08.765 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph-authtool: AQAkMAJmFjMGKhAA7G2/Au3+R+CvSQupfK+3Hw== 2024-03-26T02:17:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph-authtool: AQAlMAJmssztJRAAPqZ7j4cODHbXeD9rSs8w2Q== 2024-03-26T02:17:10.519 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph-authtool: AQAmMAJmqqbfGRAAqfCUPvCNpFAP7pXgmBrjVA== 2024-03-26T02:17:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:Creating initial monmap... 2024-03-26T02:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T02:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-03-26T02:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T02:17:11.798 INFO:teuthology.orchestra.run.smithi164.stderr:monmaptool for a [v2:172.21.15.164:3300,v1:172.21.15.164:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T02:17:11.798 INFO:teuthology.orchestra.run.smithi164.stderr:setting min_mon_release = octopus 2024-03-26T02:17:11.799 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/monmaptool: set fsid to e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:11.799 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T02:17:11.799 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:17:11.799 INFO:teuthology.orchestra.run.smithi164.stderr:Creating mon... 2024-03-26T02:17:12.745 INFO:teuthology.orchestra.run.smithi164.stderr:create mon.a on 2024-03-26T02:17:12.950 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T02:17:13.097 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715.target. 2024-03-26T02:17:13.097 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715.target. 2024-03-26T02:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Failed to reset failed state of unit ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T02:17:13.402 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715.target.wants/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@.service. 2024-03-26T02:17:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:13 smithi164 podman[24488]: 2024-03-26 02:17:13.990459553 +0000 UTC m=+0.313971794 container start 1f2545d73c977d87dae1f3f62d2f38cf0480914785b4c256de3b47da6d4a00de (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, version=8, release=754, RELEASE=HEAD, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-26T02:17:14.058 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: disabled 2024-03-26T02:17:14.065 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: inactive 2024-03-26T02:17:14.066 INFO:teuthology.orchestra.run.smithi164.stderr:firewalld.service is not enabled 2024-03-26T02:17:14.066 INFO:teuthology.orchestra.run.smithi164.stderr:Not possible to enable service . firewalld.service is not available 2024-03-26T02:17:14.066 INFO:teuthology.orchestra.run.smithi164.stderr:Waiting for mon to start... 2024-03-26T02:17:14.066 INFO:teuthology.orchestra.run.smithi164.stderr:Waiting for mon... 2024-03-26T02:17:14.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:14 smithi164 bash[24488]: 1f2545d73c977d87dae1f3f62d2f38cf0480914785b4c256de3b47da6d4a00de 2024-03-26T02:17:14.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:14 smithi164 systemd[1]: Started Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:17:14.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:14 smithi164 ceph-mon[24507]: mkfs e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:14.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:14 smithi164 ceph-mon[24507]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:17:14.547 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: cluster: 2024-03-26T02:17:14.547 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: id: e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:14.547 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-03-26T02:17:14.547 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: services: 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.495367s) 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mgr: no daemons active 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: data: 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: pgs: 2024-03-26T02:17:14.548 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:15.072 INFO:teuthology.orchestra.run.smithi164.stderr:mon is available 2024-03-26T02:17:15.072 INFO:teuthology.orchestra.run.smithi164.stderr:Assimilating anything we can from ceph.conf... 2024-03-26T02:17:15.380 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:15 smithi164 ceph-mon[24507]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:17:15.380 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:15 smithi164 ceph-mon[24507]: monmap e1: 1 mons at {a=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} 2024-03-26T02:17:15.380 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:15 smithi164 ceph-mon[24507]: fsmap 2024-03-26T02:17:15.380 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:15 smithi164 ceph-mon[24507]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T02:17:15.380 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:15 smithi164 ceph-mon[24507]: mgrmap e1: no daemons active 2024-03-26T02:17:15.380 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:15 smithi164 ceph-mon[24507]: from='client.? 172.21.15.164:0/2813001014' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:17:15.554 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:15.554 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: [global] 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: fsid = e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.164:3300,v1:172.21.15.164:6789] 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: [mgr] 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: [osd] 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-26T02:17:15.555 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-26T02:17:16.036 INFO:teuthology.orchestra.run.smithi164.stderr:Generating new minimal ceph.conf... 2024-03-26T02:17:16.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:16 smithi164 ceph-mon[24507]: from='client.? 172.21.15.164:0/482714978' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T02:17:16.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:16 smithi164 ceph-mon[24507]: from='client.? 172.21.15.164:0/482714978' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T02:17:17.010 INFO:teuthology.orchestra.run.smithi164.stderr:Restarting the monitor... 2024-03-26T02:17:17.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 systemd[1]: Stopping Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:17:17.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 bash[24726]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon.a" found: no such container 2024-03-26T02:17:17.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 ceph-mon[24507]: from='client.? 172.21.15.164:0/2825991783' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:17:17.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a[24503]: 2024-03-26T02:17:17.142+0000 7f9bc9a95700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:17:17.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a[24503]: 2024-03-26T02:17:17.142+0000 7f9bc9a95700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T02:17:17.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 podman[24737]: 2024-03-26 02:17:17.183540298 +0000 UTC m=+0.123822922 container died 1f2545d73c977d87dae1f3f62d2f38cf0480914785b4c256de3b47da6d4a00de (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, maintainer=Guillaume Abrioux , release=754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_BRANCH=HEAD, ceph=True, name=centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream) 2024-03-26T02:17:17.702 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 podman[24737]: 2024-03-26 02:17:17.500254111 +0000 UTC m=+0.440536733 container cleanup 1f2545d73c977d87dae1f3f62d2f38cf0480914785b4c256de3b47da6d4a00de (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, name=centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, architecture=x86_64, io.openshift.expose-services=, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD) 2024-03-26T02:17:17.702 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 bash[24737]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a 2024-03-26T02:17:17.702 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 bash[24763]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon.a" found: no such container 2024-03-26T02:17:18.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 podman[24752]: 2024-03-26 02:17:17.701385384 +0000 UTC m=+0.515837702 container remove 1f2545d73c977d87dae1f3f62d2f38cf0480914785b4c256de3b47da6d4a00de (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.expose-services=, name=centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-26T02:17:18.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T02:17:18.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 systemd[1]: Stopped Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:17:18.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 systemd[1]: Starting Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:17:18.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 podman[24832]: 2024-03-26 02:17:17.923465822 +0000 UTC m=+0.105576087 container create 664b72079ac2f4a96b2b33346b246e146fa3934fe6fa63dfc8df6c364bb84a4c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, RELEASE=HEAD, release=754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vcs-type=git, maintainer=Guillaume Abrioux , ceph=True, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vendor=Red Hat, Inc., name=centos-stream) 2024-03-26T02:17:18.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:17 smithi164 podman[24832]: 2024-03-26 02:17:17.835489156 +0000 UTC m=+0.017599391 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-03-26T02:17:18.208 INFO:teuthology.orchestra.run.smithi164.stderr:Setting mon public_network to 172.21.0.0/20 2024-03-26T02:17:18.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 podman[24832]: 2024-03-26 02:17:18.146008014 +0000 UTC m=+0.328118237 container init 664b72079ac2f4a96b2b33346b246e146fa3934fe6fa63dfc8df6c364bb84a4c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vcs-type=git, vendor=Red Hat, Inc., name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, architecture=x86_64) 2024-03-26T02:17:18.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 podman[24832]: 2024-03-26 02:17:18.148893139 +0000 UTC m=+0.331003354 container start 664b72079ac2f4a96b2b33346b246e146fa3934fe6fa63dfc8df6c364bb84a4c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vendor=Red Hat, Inc., release=754, distribution-scope=public, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , architecture=x86_64, vcs-type=git, io.buildah.version=1.19.8, version=8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870) 2024-03-26T02:17:18.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:17:18.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-03-26T02:17:18.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: pidfile_write: ignore empty --pid-file 2024-03-26T02:17:18.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: load: jerasure load: lrc 2024-03-26T02:17:18.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: RocksDB version: 6.15.5 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Compile date Apr 18 2022 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: DB SUMMARY 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: DB Session ID: 4GBW0GULC1YWEJ63BTEH 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82657 ; 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.env: 0x55c035ac6860 2024-03-26T02:17:18.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.fs: Posix File System 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.info_log: 0x55c0363afdc0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.statistics: (nil) 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.use_fsync: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.db_log_dir: 2024-03-26T02:17:18.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.write_buffer_manager: 0x55c0364a0240 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:17:18.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.unordered_write: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.row_cache: None 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.wal_filter: None 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.preserve_deletes: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:17:18.299 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:17:18.300 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_open_files: -1 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Compression algorithms supported: 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: kZSTD supported: 0 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:17:18.301 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.merge_operator: 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_filter: None 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c03637bd00) 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:17:18.302 INFO:journalctl@ceph.mon.a.smithi164.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: index_type: 0 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: data_block_index_type: 0 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: index_shortening: 1 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: hash_index_allow_collision: 1 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: checksum: 1 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: no_block_cache: 0 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_cache: 0x55c0363e6170 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_cache_options: 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: capacity : 536870912 2024-03-26T02:17:18.303 INFO:journalctl@ceph.mon.a.smithi164.stdout: num_shard_bits : 4 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: strict_capacity_limit : 0 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_cache_compressed: (nil) 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: persistent_cache: (nil) 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_size: 4096 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_size_deviation: 10 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_restart_interval: 16 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: index_block_restart_interval: 1 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: metadata_block_size: 4096 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: partition_filters: 0 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: use_delta_encoding: 1 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: whole_key_filtering: 1 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: verify_compression: 0 2024-03-26T02:17:18.304 INFO:journalctl@ceph.mon.a.smithi164.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout: format_version: 4 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout: enable_index_compression: 1 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_align: 0 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compression: NoCompression 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.num_levels: 7 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:17:18.305 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:17:18.306 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:17:18.307 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:17:18.307 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:17:18.307 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:17:18.307 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:17:18.307 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:17:18.307 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.arena_block_size: 4194304 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:17:18.308 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.table_properties_collectors: 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:17:18.309 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.ttl: 2592000 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-03-26T02:17:18.310 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711419438174264, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711419438175402, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79207, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77470, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71779, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711419438, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "400de2b8-7a86-4219-b836-e9b955f4ef23", "db_session_id": "4GBW0GULC1YWEJ63BTEH"}} 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711419438175880, "job": 1, "event": "recovery_finished"} 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55c0363cca80 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: DB pointer 0x55c0363dc000 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** DB Stats ** 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:17:18.311 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.8 0.00 0.00 1 0.001 0 0 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.8 0.00 0.00 1 0.001 0 0 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.8 0.00 0.00 1 0.001 0 0 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 93.8 0.00 0.00 1 0.001 0 0 2024-03-26T02:17:18.312 INFO:journalctl@ceph.mon.a.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative compaction: 0.00 GB write, 17.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval compaction: 0.00 GB write, 17.79 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.8 0.00 0.00 1 0.001 0 0 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 93.8 0.00 0.00 1 0.001 0 0 2024-03-26T02:17:18.313 INFO:journalctl@ceph.mon.a.smithi164.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 93.8 0.00 0.00 1 0.001 0 0 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative compaction: 0.00 GB write, 17.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:17:18.314 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: starting mon.a rank 0 at public addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] at bind addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a@-1(???) e1 preinit fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a@-1(???).mds e1 new map 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a@-1(???).mds e1 print_map 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout: e1 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout: legacy client fscid: -1 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout: No filesystems configured 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T02:17:18.315 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 bash[24832]: 664b72079ac2f4a96b2b33346b246e146fa3934fe6fa63dfc8df6c364bb84a4c 2024-03-26T02:17:18.316 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 systemd[1]: Started Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:17:18.316 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T02:17:18.316 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: monmap e1: 1 mons at {a=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0]} 2024-03-26T02:17:18.316 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: fsmap 2024-03-26T02:17:18.316 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T02:17:18.316 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:18 smithi164 ceph-mon[24851]: mgrmap e1: no daemons active 2024-03-26T02:17:19.373 INFO:teuthology.orchestra.run.smithi164.stderr:Wrote config to /etc/ceph/ceph.conf 2024-03-26T02:17:19.373 INFO:teuthology.orchestra.run.smithi164.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:17:19.374 INFO:teuthology.orchestra.run.smithi164.stderr:Creating mgr... 2024-03-26T02:17:19.374 INFO:teuthology.orchestra.run.smithi164.stderr:Verifying port 9283 ... 2024-03-26T02:17:19.545 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Failed to reset failed state of unit ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T02:17:19.552 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: Created symlink /etc/systemd/system/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715.target.wants/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@.service. 2024-03-26T02:17:19.987 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:19 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/4157354854' entity='client.admin' 2024-03-26T02:17:20.212 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: disabled 2024-03-26T02:17:20.219 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: inactive 2024-03-26T02:17:20.219 INFO:teuthology.orchestra.run.smithi164.stderr:firewalld.service is not enabled 2024-03-26T02:17:20.219 INFO:teuthology.orchestra.run.smithi164.stderr:Not possible to enable service . firewalld.service is not available 2024-03-26T02:17:20.229 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: disabled 2024-03-26T02:17:20.238 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: inactive 2024-03-26T02:17:20.238 INFO:teuthology.orchestra.run.smithi164.stderr:firewalld.service is not enabled 2024-03-26T02:17:20.239 INFO:teuthology.orchestra.run.smithi164.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-26T02:17:20.239 INFO:teuthology.orchestra.run.smithi164.stderr:Waiting for mgr to start... 2024-03-26T02:17:20.239 INFO:teuthology.orchestra.run.smithi164.stderr:Waiting for mgr... 2024-03-26T02:17:20.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:20 smithi164 systemd[1]: Started Ceph mgr.y for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:17:20.949 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:20.598+0000 7fa6f8fe1000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:17:20.949 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:20.687+0000 7fa6f8fe1000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:17:21.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:20 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1860851662' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:17:21.605 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:21.286+0000 7fa6f8fe1000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:17:21.605 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:21.374+0000 7fa6f8fe1000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:17:21.605 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:21.456+0000 7fa6f8fe1000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:17:21.631 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:21.631 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: { 2024-03-26T02:17:21.631 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "fsid": "e2654d3c-eb16-11ee-95d0-87774f69a715", 2024-03-26T02:17:21.631 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "health": { 2024-03-26T02:17:21.631 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 0 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: ], 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "a" 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: ], 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T02:17:21.632 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T02:17:21.633 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T02:17:21.633 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:21.633 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T02:17:21.633 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T02:17:21.633 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T02:17:21.633 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T02:17:21.633 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T02:17:21.633 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T02:17:21.636 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T02:17:21.636 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T02:17:21.636 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T02:17:21.637 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "available": false, 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "iostat", 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "nfs", 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "restful" 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: ], 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "services": {} 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T02:17:21.638 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "modified": "2024-03-26T02:17:14.051604+0000", 2024-03-26T02:17:21.639 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "services": {} 2024-03-26T02:17:21.639 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:21.639 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T02:17:21.639 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: } 2024-03-26T02:17:21.880 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:21.604+0000 7fa6f8fe1000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:17:21.880 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:21.685+0000 7fa6f8fe1000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:17:21.881 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:21.795+0000 7fa6f8fe1000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:17:21.881 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:21.879+0000 7fa6f8fe1000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:17:22.133 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:22.045+0000 7fa6f8fe1000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:17:22.133 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:22.132+0000 7fa6f8fe1000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:17:23.512 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:23.193+0000 7fa6f8fe1000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:17:23.512 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:23.322+0000 7fa6f8fe1000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:17:23.512 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:23.511+0000 7fa6f8fe1000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:17:23.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:23.599+0000 7fa6f8fe1000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:17:23.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:23.733+0000 7fa6f8fe1000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:17:24.357 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:24.055+0000 7fa6f8fe1000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:17:24.357 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:24.273+0000 7fa6f8fe1000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:17:24.357 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:24.356+0000 7fa6f8fe1000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:17:24.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:24.526+0000 7fa6f8fe1000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:17:25.729 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:25 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:25.437+0000 7fa6f8fe1000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:17:25.729 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:25 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:25.728+0000 7fa6f8fe1000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:17:26.034 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:25 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:25.822+0000 7fa6f8fe1000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:17:26.035 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:25 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:25.970+0000 7fa6f8fe1000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:17:26.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: Activating manager daemon y 2024-03-26T02:17:26.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: mgrmap e2: y(active, starting, since 0.00270291s) 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: Manager daemon y is now available 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' 2024-03-26T02:17:26.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:26 smithi164 ceph-mon[24851]: from='mgr.14100 172.21.15.164:0/1170562256' entity='mgr.y' 2024-03-26T02:17:27.154 INFO:teuthology.orchestra.run.smithi164.stderr:mgr not available, waiting (1/15)... 2024-03-26T02:17:28.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:27 smithi164 ceph-mon[24851]: mgrmap e3: y(active, since 1.00931s) 2024-03-26T02:17:29.806 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: { 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "fsid": "e2654d3c-eb16-11ee-95d0-87774f69a715", 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "health": { 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "checks": {}, 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "mutes": [] 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "quorum": [ 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 0 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: ], 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "quorum_names": [ 2024-03-26T02:17:29.807 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "a" 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: ], 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "monmap": { 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_mons": 1 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "osdmap": { 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_osds": 0, 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-03-26T02:17:29.808 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "pgmap": { 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_pools": 0, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_objects": 0, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-03-26T02:17:29.812 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "bytes_total": 0 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "fsmap": { 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "by_rank": [], 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "up:standby": 0 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "mgrmap": { 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "available": true, 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "modules": [ 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "iostat", 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "nfs", 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "restful" 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: ], 2024-03-26T02:17:29.813 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "services": {} 2024-03-26T02:17:29.814 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:29.814 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "servicemap": { 2024-03-26T02:17:29.814 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 1, 2024-03-26T02:17:29.814 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "modified": "2024-03-26T02:17:14.051604+0000", 2024-03-26T02:17:29.814 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "services": {} 2024-03-26T02:17:29.814 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: }, 2024-03-26T02:17:29.814 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "progress_events": {} 2024-03-26T02:17:29.814 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: } 2024-03-26T02:17:30.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:29 smithi164 ceph-mon[24851]: mgrmap e4: y(active, since 3s) 2024-03-26T02:17:30.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:29 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1628861891' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T02:17:30.357 INFO:teuthology.orchestra.run.smithi164.stderr:mgr is available 2024-03-26T02:17:30.897 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:30.897 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: [global] 2024-03-26T02:17:30.897 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: fsid = e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:30.897 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-03-26T02:17:30.897 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-03-26T02:17:30.897 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-03-26T02:17:30.897 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-03-26T02:17:30.898 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:30.898 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: [mgr] 2024-03-26T02:17:30.898 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-03-26T02:17:30.898 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 2024-03-26T02:17:30.898 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: [osd] 2024-03-26T02:17:30.898 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-03-26T02:17:30.898 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-03-26T02:17:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:30 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1071878682' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T02:17:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:30 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1071878682' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T02:17:31.469 INFO:teuthology.orchestra.run.smithi164.stderr:Enabling cephadm module... 2024-03-26T02:17:32.413 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:32 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2990010947' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T02:17:32.413 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ignoring --setuser ceph since I am not root 2024-03-26T02:17:32.414 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ignoring --setgroup ceph since I am not root 2024-03-26T02:17:32.688 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:32.598+0000 7f5420c34000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:17:32.689 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:32.687+0000 7f5420c34000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:17:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2990010947' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T02:17:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-mon[24851]: mgrmap e5: y(active, since 6s) 2024-03-26T02:17:33.330 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: { 2024-03-26T02:17:33.331 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 5, 2024-03-26T02:17:33.331 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "available": true, 2024-03-26T02:17:33.331 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-26T02:17:33.331 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-26T02:17:33.331 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: } 2024-03-26T02:17:33.626 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:33.297+0000 7f5420c34000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:17:33.627 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:33.391+0000 7f5420c34000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:17:33.627 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:33.474+0000 7f5420c34000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:17:33.627 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:33.625+0000 7f5420c34000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:17:33.902 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:33.707+0000 7f5420c34000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:17:33.902 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:33.816+0000 7f5420c34000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:17:33.902 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:33.900+0000 7f5420c34000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:17:34.185 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:34 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:34.064+0000 7f5420c34000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:17:34.185 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:34 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:34.149+0000 7f5420c34000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:17:34.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:34 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3860752344' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T02:17:35.099 INFO:teuthology.orchestra.run.smithi164.stderr:Waiting for the mgr to restart... 2024-03-26T02:17:35.099 INFO:teuthology.orchestra.run.smithi164.stderr:Waiting for mgr epoch 5... 2024-03-26T02:17:35.505 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:35.186+0000 7f5420c34000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:17:35.506 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:35.317+0000 7f5420c34000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:17:35.506 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:35.504+0000 7f5420c34000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:17:35.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:35.590+0000 7f5420c34000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:17:35.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:35.724+0000 7f5420c34000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:17:36.351 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:36 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:36.045+0000 7f5420c34000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:17:36.351 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:36 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:36.266+0000 7f5420c34000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:17:36.351 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:36 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:36.349+0000 7f5420c34000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:17:36.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:36 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:36.518+0000 7f5420c34000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:17:37.735 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:37 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:37.442+0000 7f5420c34000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:17:37.735 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:37 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:37.733+0000 7f5420c34000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:17:38.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:37 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:37.828+0000 7f5420c34000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:17:38.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:37 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:37.981+0000 7f5420c34000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:17:39.057 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: { 2024-03-26T02:17:39.057 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-03-26T02:17:39.057 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "initialized": true 2024-03-26T02:17:39.057 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: } 2024-03-26T02:17:39.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:17:38] ENGINE Bus STARTING 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:17:38] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:17:38] ENGINE Bus STARTED 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: Active manager daemon y restarted 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: Activating manager daemon y 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: mgrmap e6: y(active, starting, since 0.0529692s) 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:17:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: Manager daemon y is now available 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:17:39.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:38 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:39.589 INFO:teuthology.orchestra.run.smithi164.stderr:mgr epoch 5 is available 2024-03-26T02:17:39.589 INFO:teuthology.orchestra.run.smithi164.stderr:Setting orchestrator backend to cephadm... 2024-03-26T02:17:40.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:39 smithi164 ceph-mon[24851]: [26/Mar/2024:02:17:38] ENGINE Bus STARTING 2024-03-26T02:17:40.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:39 smithi164 ceph-mon[24851]: [26/Mar/2024:02:17:38] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:17:40.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:39 smithi164 ceph-mon[24851]: [26/Mar/2024:02:17:38] ENGINE Bus STARTED 2024-03-26T02:17:40.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:39 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:40.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:39 smithi164 ceph-mon[24851]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T02:17:40.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:39 smithi164 ceph-mon[24851]: mgrmap e7: y(active, since 1.05484s) 2024-03-26T02:17:40.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:39 smithi164 ceph-mon[24851]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T02:17:41.199 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: value unchanged 2024-03-26T02:17:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:41 smithi164 ceph-mon[24851]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:17:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:41 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:41 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:41 smithi164 ceph-mon[24851]: mgrmap e8: y(active, since 3s) 2024-03-26T02:17:41.701 INFO:teuthology.orchestra.run.smithi164.stderr:Generating ssh key... 2024-03-26T02:17:42.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-mon[24851]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: Generating public/private rsa key pair. 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: Your identification has been saved in /tmp/tmpeh9ac62n/key. 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: Your public key has been saved in /tmp/tmpeh9ac62n/key.pub. 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: The key fingerprint is: 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: SHA256:SUn7gVvqUkAGUG0VgviROvb43QThlkz3Sje0tx6WPEc ceph-e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: The key's randomart image is: 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: +---[RSA 3072]----+ 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | .+o=+.+. | 2024-03-26T02:17:42.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | . +o=o.+. | 2024-03-26T02:17:42.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | o *.+=oo. | 2024-03-26T02:17:42.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | + . *o.*=.. E | 2024-03-26T02:17:42.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | . + . oSo.+ + | 2024-03-26T02:17:42.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | . . oo B . | 2024-03-26T02:17:42.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | . ..o. o + | 2024-03-26T02:17:42.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | . ... . | 2024-03-26T02:17:42.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: | | 2024-03-26T02:17:42.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: +----[SHA256]-----+ 2024-03-26T02:17:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4iLZok3jUCx1mQXh9hrY1kHjSLiI5xpavjnRl75xSLl1rb12M6FDrnPIOLSfgjaF8Ynv4+pGaQFBYeeiS17SwysBrGTGhMia58xXM2Vk5aqvEe+Lm6SZFHCF7V1EZlT5LHNkcR2BQ4hNHIsKH+SDT21hyk15KlLfN4thLD7dGQUOCEJeHT6B6+TOGIorRfscmxkjsQx35m4Wc4drmYGpEusgCqJMl7cDfQwMGMxCPc7Lz53oCqECFN14JVB6L9QzAJTk/4JCvtUV/NpCgfamUjKPnGZnGuSDrh52CUs9Zw+IEaCk5QKE7rvaZiY5dBaX61P2TjEgx96riXeINpaydCcPP4U4xlSS47ZGooCrRsavTVwnrpQ78rWv+75DodZipeh9bbydNP1u3JKbhYf8WEzDjs30gIhF6GqgawlguYtOLWWMFd8+hVMbD5yEWtbaUlYQ5U3Wcf9qVhAD8Cx447clA2kU4iRjIe2g4BPyEPMI+Kseyp3lAHmbkWz/NN1U= ceph-e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:17:43.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:43 smithi164 ceph-mon[24851]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:17:43.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:43 smithi164 ceph-mon[24851]: Generating ssh key... 2024-03-26T02:17:43.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:43 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:43.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:43 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:17:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:Adding key to root@localhost authorized_keys... 2024-03-26T02:17:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:Adding host smithi164... 2024-03-26T02:17:44.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:44 smithi164 ceph-mon[24851]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:17:45.559 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:45 smithi164 ceph-mon[24851]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi164", "addr": "172.21.15.164", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:17:45.560 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:45 smithi164 ceph-mon[24851]: Deploying cephadm binary to smithi164 2024-03-26T02:17:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: Added host 'smithi164' with addr '172.21.15.164' 2024-03-26T02:17:46.188 INFO:teuthology.orchestra.run.smithi164.stderr:Deploying unmanaged mon service... 2024-03-26T02:17:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:46 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:46.842 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:46 smithi164 ceph-mon[24851]: Added host smithi164 2024-03-26T02:17:46.843 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:46 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:46.854 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: Scheduled mon update... 2024-03-26T02:17:47.462 INFO:teuthology.orchestra.run.smithi164.stderr:Deploying unmanaged mgr service... 2024-03-26T02:17:48.119 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:47 smithi164 ceph-mon[24851]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:17:48.119 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:47 smithi164 ceph-mon[24851]: Saving service mon spec with placement count:5 2024-03-26T02:17:48.119 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:47 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-03-26T02:17:49.491 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:49 smithi164 ceph-mon[24851]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:17:49.491 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:49 smithi164 ceph-mon[24851]: Saving service mgr spec with placement count:2 2024-03-26T02:17:49.491 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:49 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:49.492 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:49 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:50.925 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:50 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:50.925 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:50 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2243357560' entity='client.admin' 2024-03-26T02:17:52.427 INFO:teuthology.orchestra.run.smithi164.stderr:Enabling the dashboard module... 2024-03-26T02:17:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:52 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:52 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1717572404' entity='client.admin' 2024-03-26T02:17:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:52 smithi164 ceph-mon[24851]: from='mgr.14116 172.21.15.164:0/3044431235' entity='mgr.y' 2024-03-26T02:17:53.456 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:53 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1608789701' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T02:17:53.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ignoring --setuser ceph since I am not root 2024-03-26T02:17:53.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ignoring --setgroup ceph since I am not root 2024-03-26T02:17:54.218 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:53.934+0000 7fea67ac9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:17:54.218 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:54.024+0000 7fea67ac9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:17:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:54 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1608789701' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T02:17:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:54 smithi164 ceph-mon[24851]: mgrmap e9: y(active, since 15s) 2024-03-26T02:17:54.879 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:54.615+0000 7fea67ac9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:17:54.879 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:54.704+0000 7fea67ac9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:17:54.879 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:54.790+0000 7fea67ac9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:17:55.132 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:54.940+0000 7fea67ac9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:17:55.132 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:55 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:55.024+0000 7fea67ac9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:17:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: { 2024-03-26T02:17:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "epoch": 9, 2024-03-26T02:17:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "available": true, 2024-03-26T02:17:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "active_name": "y", 2024-03-26T02:17:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "num_standby": 0 2024-03-26T02:17:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: } 2024-03-26T02:17:55.401 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:55 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:55.138+0000 7fea67ac9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:17:55.401 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:55 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:55.223+0000 7fea67ac9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:17:55.401 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:55 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:55.399+0000 7fea67ac9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:17:55.711 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:55 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/939571781' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T02:17:55.712 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:55 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:55.485+0000 7fea67ac9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:17:55.997 INFO:teuthology.orchestra.run.smithi164.stderr:Waiting for the mgr to restart... 2024-03-26T02:17:55.998 INFO:teuthology.orchestra.run.smithi164.stderr:Waiting for mgr epoch 9... 2024-03-26T02:17:56.649 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:56 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:56.520+0000 7fea67ac9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:17:56.649 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:56 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:56.647+0000 7fea67ac9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:17:56.921 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:56 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:56.834+0000 7fea67ac9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:17:56.921 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:56 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:56.919+0000 7fea67ac9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:17:57.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:57.052+0000 7fea67ac9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:17:57.669 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:57.368+0000 7fea67ac9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:17:57.669 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:57.584+0000 7fea67ac9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:17:57.669 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:57.667+0000 7fea67ac9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:17:58.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:57.834+0000 7fea67ac9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:17:59.037 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:58.743+0000 7fea67ac9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:17:59.038 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:59.036+0000 7fea67ac9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:17:59.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:59.130+0000 7fea67ac9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:17:59.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:17:59.281+0000 7fea67ac9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:17:59.602 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-mon[24851]: Active manager daemon y restarted 2024-03-26T02:17:59.603 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-mon[24851]: Activating manager daemon y 2024-03-26T02:17:59.603 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-mon[24851]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T02:17:59.603 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:17:59] ENGINE Bus STARTING 2024-03-26T02:18:00.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:17:59] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:18:00.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:17:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:17:59] ENGINE Bus STARTED 2024-03-26T02:18:00.363 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: { 2024-03-26T02:18:00.363 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-03-26T02:18:00.363 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: "initialized": true 2024-03-26T02:18:00.364 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: } 2024-03-26T02:18:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: mgrmap e10: y(active, starting, since 0.0551683s) 2024-03-26T02:18:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: Manager daemon y is now available 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:18:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:01.191 INFO:teuthology.orchestra.run.smithi164.stderr:mgr epoch 9 is available 2024-03-26T02:18:01.192 INFO:teuthology.orchestra.run.smithi164.stderr:Generating a dashboard self-signed certificate... 2024-03-26T02:18:01.534 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:01 smithi164 ceph-mon[24851]: [26/Mar/2024:02:17:59] ENGINE Bus STARTING 2024-03-26T02:18:01.535 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:01 smithi164 ceph-mon[24851]: [26/Mar/2024:02:17:59] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:18:01.535 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:01 smithi164 ceph-mon[24851]: [26/Mar/2024:02:17:59] ENGINE Bus STARTED 2024-03-26T02:18:01.535 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:01 smithi164 ceph-mon[24851]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T02:18:01.535 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:01 smithi164 ceph-mon[24851]: mgrmap e11: y(active, since 1.06119s) 2024-03-26T02:18:01.535 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:01 smithi164 ceph-mon[24851]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T02:18:01.535 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:01 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: Self-signed certificate created 2024-03-26T02:18:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:02 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:02 smithi164 ceph-mon[24851]: mgrmap e12: y(active, since 2s) 2024-03-26T02:18:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:02 smithi164 ceph-mon[24851]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:18:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:02 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:02 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:02.968 INFO:teuthology.orchestra.run.smithi164.stderr:Creating initial admin user... 2024-03-26T02:18:03.875 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$aneEsajWWl1TKDgmYjvn/umg2ib6iBgDFg01mfxhq2K4zBs8m9EDy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711419483, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T02:18:04.554 INFO:teuthology.orchestra.run.smithi164.stderr:Fetching dashboard port number... 2024-03-26T02:18:05.217 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:04 smithi164 ceph-mon[24851]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:18:05.217 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:04 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:05.217 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: 8443 2024-03-26T02:18:05.871 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: disabled 2024-03-26T02:18:05.879 INFO:teuthology.orchestra.run.smithi164.stderr:systemctl: inactive 2024-03-26T02:18:05.880 INFO:teuthology.orchestra.run.smithi164.stderr:firewalld.service is not enabled 2024-03-26T02:18:05.880 INFO:teuthology.orchestra.run.smithi164.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T02:18:05.883 INFO:teuthology.orchestra.run.smithi164.stderr:Ceph Dashboard is now available at: 2024-03-26T02:18:05.883 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:18:05.883 INFO:teuthology.orchestra.run.smithi164.stderr: URL: https://smithi164.front.sepia.ceph.com:8443/ 2024-03-26T02:18:05.883 INFO:teuthology.orchestra.run.smithi164.stderr: User: admin 2024-03-26T02:18:05.883 INFO:teuthology.orchestra.run.smithi164.stderr: Password: cm5yuklk0h 2024-03-26T02:18:05.883 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:18:05.883 INFO:teuthology.orchestra.run.smithi164.stderr:Enabling autotune for osd_memory_target 2024-03-26T02:18:06.166 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:05 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2540432460' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T02:18:07.240 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:06 smithi164 ceph-mon[24851]: mgrmap e13: y(active, since 6s) 2024-03-26T02:18:07.983 INFO:teuthology.orchestra.run.smithi164.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-03-26T02:18:08.434 INFO:teuthology.orchestra.run.smithi164.stderr:You can access the Ceph CLI with: 2024-03-26T02:18:08.434 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr:Please consider enabling telemetry to help improve Ceph: 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr: ceph telemetry on 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr:For more information see: 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T02:18:08.435 INFO:teuthology.orchestra.run.smithi164.stderr:Bootstrap complete. 2024-03-26T02:18:08.472 INFO:tasks.cephadm:Fetching config... 2024-03-26T02:18:08.473 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:18:08.473 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T02:18:08.489 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T02:18:08.489 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:18:08.489 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T02:18:08.544 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T02:18:08.544 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:18:08.544 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T02:18:08.612 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T02:18:08.612 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:18:08.613 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T02:18:08.667 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T02:18:08.667 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4iLZok3jUCx1mQXh9hrY1kHjSLiI5xpavjnRl75xSLl1rb12M6FDrnPIOLSfgjaF8Ynv4+pGaQFBYeeiS17SwysBrGTGhMia58xXM2Vk5aqvEe+Lm6SZFHCF7V1EZlT5LHNkcR2BQ4hNHIsKH+SDT21hyk15KlLfN4thLD7dGQUOCEJeHT6B6+TOGIorRfscmxkjsQx35m4Wc4drmYGpEusgCqJMl7cDfQwMGMxCPc7Lz53oCqECFN14JVB6L9QzAJTk/4JCvtUV/NpCgfamUjKPnGZnGuSDrh52CUs9Zw+IEaCk5QKE7rvaZiY5dBaX61P2TjEgx96riXeINpaydCcPP4U4xlSS47ZGooCrRsavTVwnrpQ78rWv+75DodZipeh9bbydNP1u3JKbhYf8WEzDjs30gIhF6GqgawlguYtOLWWMFd8+hVMbD5yEWtbaUlYQ5U3Wcf9qVhAD8Cx447clA2kU4iRjIe2g4BPyEPMI+Kseyp3lAHmbkWz/NN1U= ceph-e2654d3c-eb16-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T02:18:08.760 INFO:teuthology.orchestra.run.smithi164.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4iLZok3jUCx1mQXh9hrY1kHjSLiI5xpavjnRl75xSLl1rb12M6FDrnPIOLSfgjaF8Ynv4+pGaQFBYeeiS17SwysBrGTGhMia58xXM2Vk5aqvEe+Lm6SZFHCF7V1EZlT5LHNkcR2BQ4hNHIsKH+SDT21hyk15KlLfN4thLD7dGQUOCEJeHT6B6+TOGIorRfscmxkjsQx35m4Wc4drmYGpEusgCqJMl7cDfQwMGMxCPc7Lz53oCqECFN14JVB6L9QzAJTk/4JCvtUV/NpCgfamUjKPnGZnGuSDrh52CUs9Zw+IEaCk5QKE7rvaZiY5dBaX61P2TjEgx96riXeINpaydCcPP4U4xlSS47ZGooCrRsavTVwnrpQ78rWv+75DodZipeh9bbydNP1u3JKbhYf8WEzDjs30gIhF6GqgawlguYtOLWWMFd8+hVMbD5yEWtbaUlYQ5U3Wcf9qVhAD8Cx447clA2kU4iRjIe2g4BPyEPMI+Kseyp3lAHmbkWz/NN1U= ceph-e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:18:08.775 DEBUG:teuthology.orchestra.run.smithi172:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4iLZok3jUCx1mQXh9hrY1kHjSLiI5xpavjnRl75xSLl1rb12M6FDrnPIOLSfgjaF8Ynv4+pGaQFBYeeiS17SwysBrGTGhMia58xXM2Vk5aqvEe+Lm6SZFHCF7V1EZlT5LHNkcR2BQ4hNHIsKH+SDT21hyk15KlLfN4thLD7dGQUOCEJeHT6B6+TOGIorRfscmxkjsQx35m4Wc4drmYGpEusgCqJMl7cDfQwMGMxCPc7Lz53oCqECFN14JVB6L9QzAJTk/4JCvtUV/NpCgfamUjKPnGZnGuSDrh52CUs9Zw+IEaCk5QKE7rvaZiY5dBaX61P2TjEgx96riXeINpaydCcPP4U4xlSS47ZGooCrRsavTVwnrpQ78rWv+75DodZipeh9bbydNP1u3JKbhYf8WEzDjs30gIhF6GqgawlguYtOLWWMFd8+hVMbD5yEWtbaUlYQ5U3Wcf9qVhAD8Cx447clA2kU4iRjIe2g4BPyEPMI+Kseyp3lAHmbkWz/NN1U= ceph-e2654d3c-eb16-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T02:18:08.824 INFO:teuthology.orchestra.run.smithi172.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4iLZok3jUCx1mQXh9hrY1kHjSLiI5xpavjnRl75xSLl1rb12M6FDrnPIOLSfgjaF8Ynv4+pGaQFBYeeiS17SwysBrGTGhMia58xXM2Vk5aqvEe+Lm6SZFHCF7V1EZlT5LHNkcR2BQ4hNHIsKH+SDT21hyk15KlLfN4thLD7dGQUOCEJeHT6B6+TOGIorRfscmxkjsQx35m4Wc4drmYGpEusgCqJMl7cDfQwMGMxCPc7Lz53oCqECFN14JVB6L9QzAJTk/4JCvtUV/NpCgfamUjKPnGZnGuSDrh52CUs9Zw+IEaCk5QKE7rvaZiY5dBaX61P2TjEgx96riXeINpaydCcPP4U4xlSS47ZGooCrRsavTVwnrpQ78rWv+75DodZipeh9bbydNP1u3JKbhYf8WEzDjs30gIhF6GqgawlguYtOLWWMFd8+hVMbD5yEWtbaUlYQ5U3Wcf9qVhAD8Cx447clA2kU4iRjIe2g4BPyEPMI+Kseyp3lAHmbkWz/NN1U= ceph-e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:18:08.839 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T02:18:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:08 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/767478229' entity='client.admin' 2024-03-26T02:18:11.761 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:11.761 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi164", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:18:11.761 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:11.762 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:11.762 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:12.792 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T02:18:12.792 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T02:18:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:12 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/642493923' entity='client.admin' 2024-03-26T02:18:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:15.163 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi172 2024-03-26T02:18:15.163 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-26T02:18:15.163 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/etc/ceph/ceph.conf 2024-03-26T02:18:15.182 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-26T02:18:15.182 DEBUG:teuthology.orchestra.run.smithi172:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:18:15.240 INFO:tasks.cephadm:Adding host smithi172 to orchestrator... 2024-03-26T02:18:15.240 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch host add smithi172 2024-03-26T02:18:15.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:18:15.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:15.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:15.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:15.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:15.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:18:15.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:18:15.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:15.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:15 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:17.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:17 smithi164 ceph-mon[24851]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:18:18.131 INFO:teuthology.orchestra.run.smithi164.stdout:Added host 'smithi172' with addr '172.21.15.172' 2024-03-26T02:18:18.706 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T02:18:19.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:19 smithi164 ceph-mon[24851]: Deploying cephadm binary to smithi172 2024-03-26T02:18:19.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:19.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:19 smithi164 ceph-mon[24851]: Added host smithi172 2024-03-26T02:18:19.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:19.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:19.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:19.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:19.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:20.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:20 smithi164 ceph-mon[24851]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:20.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:20 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:20.599 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:18:20.599 INFO:teuthology.orchestra.run.smithi164.stdout:[{"addr": "172.21.15.164", "hostname": "smithi164", "labels": [], "status": ""}, {"addr": "172.21.15.172", "hostname": "smithi172", "labels": [], "status": ""}] 2024-03-26T02:18:21.175 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T02:18:21.175 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T02:18:22.659 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:22 smithi164 ceph-mon[24851]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:18:22.660 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:22 smithi164 ceph-mon[24851]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:23.598 INFO:teuthology.orchestra.run.smithi164.stderr:adjusted tunables profile to default 2024-03-26T02:18:23.896 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:23 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:23.896 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:23 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:23.896 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:23 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi172", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:18:23.896 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:23 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:23.896 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:23 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/4146670030' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T02:18:23.896 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:23 smithi164 ceph-mon[24851]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:24.157 INFO:tasks.cephadm:Adding mon.a on smithi164 2024-03-26T02:18:24.157 INFO:tasks.cephadm:Adding mon.c on smithi164 2024-03-26T02:18:24.157 INFO:tasks.cephadm:Adding mon.b on smithi172 2024-03-26T02:18:24.157 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi164:172.21.15.164=a;smithi164:[v2:172.21.15.164:3301,v1:172.21.15.164:6790]=c;smithi172:172.21.15.172=b' 2024-03-26T02:18:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:24 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/4146670030' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T02:18:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:24 smithi164 ceph-mon[24851]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:18:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:25 smithi164 ceph-mon[24851]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:26.125 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled mon update... 2024-03-26T02:18:26.789 DEBUG:teuthology.orchestra.run.smithi164:mon.c> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T02:18:26.792 DEBUG:teuthology.orchestra.run.smithi172:mon.b> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T02:18:26.794 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:18:26.794 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:18:27.432 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi164:172.21.15.164=a;smithi164:[v2:172.21.15.164:3301,v1:172.21.15.164:6790]=c;smithi172:172.21.15.172=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:18:27.433 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: Saving service mon spec with placement smithi164:172.21.15.164=a;smithi164:[v2:172.21.15.164:3301,v1:172.21.15.164:6790]=c;smithi172:172.21.15.172=b;count:3 2024-03-26T02:18:27.433 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:27.433 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:27.433 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:27.433 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:27.433 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:18:27.433 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:18:27.433 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:27.434 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:27.434 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:27.434 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:28.270 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:28 smithi164 ceph-mon[24851]: Deploying daemon mon.c on smithi164 2024-03-26T02:18:28.270 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:28 smithi164 ceph-mon[24851]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:28.758 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:18:28.759 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":1,"fsid":"e2654d3c-eb16-11ee-95d0-87774f69a715","modified":"2024-03-26T02:17:11.294549Z","created":"2024-03-26T02:17:11.294549Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T02:18:28.760 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 1 2024-03-26T02:18:29.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:29 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/2527883804' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:18:30.343 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:18:30.344 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: mon.a calling monitor election 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: mon.c calling monitor election 2024-03-26T02:18:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: monmap e2: 2 mons at {a=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],c=[v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0]} 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: fsmap 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: mgrmap e13: y(active, since 35s) 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: overall HEALTH_OK 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:35.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:35.375 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:18:35.376 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":2,"fsid":"e2654d3c-eb16-11ee-95d0-87774f69a715","modified":"2024-03-26T02:18:29.859211Z","created":"2024-03-26T02:17:11.294549Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T02:18:35.377 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 2 2024-03-26T02:18:36.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:35 smithi164 ceph-mon[29512]: Deploying daemon mon.b on smithi172 2024-03-26T02:18:36.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:35 smithi164 ceph-mon[29512]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:36.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:35 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/871655225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:18:36.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:35 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:36.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:35 smithi164 ceph-mon[24851]: Deploying daemon mon.b on smithi172 2024-03-26T02:18:36.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:35 smithi164 ceph-mon[24851]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:36.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:35 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/871655225' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:18:36.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:35 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:37.445 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:18:37.445 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:18:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:38 smithi164 ceph-mon[24851]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:38.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:38 smithi164 ceph-mon[29512]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:39.597 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:18:39.597 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":2,"fsid":"e2654d3c-eb16-11ee-95d0-87774f69a715","modified":"2024-03-26T02:18:29.859211Z","created":"2024-03-26T02:17:11.294549Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T02:18:39.597 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 2 2024-03-26T02:18:41.483 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T02:18:41.484 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: mon.a calling monitor election 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: mon.c calling monitor election 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: mon.b calling monitor election 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: monmap e3: 3 mons at {a=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0]} 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: fsmap 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: mgrmap e13: y(active, since 45s) 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: overall HEALTH_OK 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:45.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: mon.a calling monitor election 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: mon.c calling monitor election 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: mon.b calling monitor election 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:18:45.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: monmap e3: 3 mons at {a=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0]} 2024-03-26T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: fsmap 2024-03-26T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: mgrmap e13: y(active, since 45s) 2024-03-26T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: overall HEALTH_OK 2024-03-26T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:45.298 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:45 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:46.223 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:46 smithi164 ceph-mon[24851]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:46.223 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:46 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:46.223 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:46 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:46.223 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:46 smithi164 ceph-mon[29512]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:46 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:18:46.224 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:46 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:46.367 INFO:teuthology.orchestra.run.smithi172.stdout: 2024-03-26T02:18:46.368 INFO:teuthology.orchestra.run.smithi172.stdout:{"epoch":3,"fsid":"e2654d3c-eb16-11ee-95d0-87774f69a715","modified":"2024-03-26T02:18:39.946932Z","created":"2024-03-26T02:17:11.294549Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:3300","nonce":0},{"type":"v1","addr":"172.21.15.172:6789","nonce":0}]},"addr":"172.21.15.172:6789/0","public_addr":"172.21.15.172:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T02:18:46.369 INFO:teuthology.orchestra.run.smithi172.stderr:dumped monmap epoch 3 2024-03-26T02:18:46.897 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T02:18:46.898 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/332490778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:18:47.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[24851]: Reconfiguring daemon mon.a on smithi164 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/332490778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:18:47.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:47.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:47 smithi164 ceph-mon[29512]: Reconfiguring daemon mon.a on smithi164 2024-03-26T02:18:47.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:47 smithi172 ceph-mon[25688]: Reconfiguring daemon mon.a on smithi164 2024-03-26T02:18:48.500 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[29512]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:48.500 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:48.500 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:48.500 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:18:48.500 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:48.501 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[24851]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:48.501 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:48.501 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:48.501 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:18:48.501 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:48 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:48.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:48 smithi172 ceph-mon[25688]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:48.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:48 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:48.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:48 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:48.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:48 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:18:48.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:48 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:49.389 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:49 smithi164 ceph-mon[24851]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:18:49.389 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:49 smithi164 ceph-mon[24851]: Reconfiguring daemon mon.c on smithi164 2024-03-26T02:18:49.389 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:49 smithi164 ceph-mon[29512]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:18:49.389 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:49 smithi164 ceph-mon[29512]: Reconfiguring daemon mon.c on smithi164 2024-03-26T02:18:49.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:49 smithi172 ceph-mon[25688]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:18:49.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:49 smithi172 ceph-mon[25688]: Reconfiguring daemon mon.c on smithi164 2024-03-26T02:18:50.115 INFO:teuthology.orchestra.run.smithi164.stdout:# minimal ceph.conf for e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:18:50.115 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-03-26T02:18:50.116 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:18:50.116 INFO:teuthology.orchestra.run.smithi164.stdout: mon_host = [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-03-26T02:18:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:50 smithi164 ceph-mon[24851]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:50 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:50 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/158360341' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:50.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:50 smithi164 ceph-mon[29512]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:50 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:50 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/158360341' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:50 smithi172 ceph-mon[25688]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:50 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:50 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/158360341' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:51.021 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T02:18:51.022 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:18:51.022 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T02:18:51.050 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:18:51.050 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:18:51.117 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-26T02:18:51.117 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T02:18:51.145 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-26T02:18:51.145 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:18:51.213 INFO:tasks.cephadm:Adding mgr.y on smithi164 2024-03-26T02:18:51.213 INFO:tasks.cephadm:Adding mgr.x on smithi172 2024-03-26T02:18:51.213 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi164=y;smithi172=x' 2024-03-26T02:18:51.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[24851]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[24851]: Reconfiguring daemon mon.b on smithi172 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[24851]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[29512]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[29512]: Reconfiguring daemon mon.b on smithi172 2024-03-26T02:18:51.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:51 smithi164 ceph-mon[29512]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:51 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:51 smithi172 ceph-mon[25688]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:18:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:51 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:18:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:51 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:18:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:51 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:51 smithi172 ceph-mon[25688]: Reconfiguring daemon mon.b on smithi172 2024-03-26T02:18:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:51 smithi172 ceph-mon[25688]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:53.084 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:52 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:53.084 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:52 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:53.084 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:52 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:53.084 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:52 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:53.085 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:52 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:53.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:52 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:53.462 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled mgr update... 2024-03-26T02:18:54.095 DEBUG:teuthology.orchestra.run.smithi172:mgr.x> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T02:18:54.098 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T02:18:54.098 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:18:54.098 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:18:54.117 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T02:18:54.117 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-03-26T02:18:54.175 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T02:18:54.175 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:18:54.175 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-26T02:18:54.175 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:18:54.175 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:18:54.175 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-26 02:18:29.267015823 +0000 2024-03-26T02:18:54.175 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-26 02:16:03.341753686 +0000 2024-03-26T02:18:54.175 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-26 02:16:03.341753686 +0000 2024-03-26T02:18:54.176 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-26 02:16:03.341753686 +0000 2024-03-26T02:18:54.176 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T02:18:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-26T02:18:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-26T02:18:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000201836 s, 2.5 MB/s 2024-03-26T02:18:54.245 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T02:18:54.302 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-03-26T02:18:54.359 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T02:18:54.359 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:18:54.359 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-26T02:18:54.359 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:18:54.359 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:18:54.359 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-26 02:18:29.268015806 +0000 2024-03-26T02:18:54.360 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-26 02:16:03.688747804 +0000 2024-03-26T02:18:54.360 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-26 02:16:03.688747804 +0000 2024-03-26T02:18:54.360 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-26 02:16:03.688747804 +0000 2024-03-26T02:18:54.360 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T02:18:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-26T02:18:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-26T02:18:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000202186 s, 2.5 MB/s 2024-03-26T02:18:54.427 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T02:18:54.483 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-26 02:18:29.268015806 +0000 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-26 02:16:04.036741906 +0000 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-26 02:16:04.036741906 +0000 2024-03-26T02:18:54.540 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-26 02:16:04.036741906 +0000 2024-03-26T02:18:54.540 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T02:18:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi164=y;smithi172=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:18:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: Saving service mgr spec with placement smithi164=y;smithi172=x;count:2 2024-03-26T02:18:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi164=y;smithi172=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: Saving service mgr spec with placement smithi164=y;smithi172=x;count:2 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:18:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:18:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:18:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-26T02:18:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-26T02:18:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000183231 s, 2.8 MB/s 2024-03-26T02:18:54.572 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T02:18:54.628 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-03-26T02:18:54.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:54.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi164=y;smithi172=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:18:54.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: Saving service mgr spec with placement smithi164=y;smithi172=x;count:2 2024-03-26T02:18:54.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:54.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:54.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:54.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:54.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:54.661 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:18:54.661 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T02:18:54.661 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:18:54.661 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:54.682 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T02:18:54.683 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:18:54.683 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T02:18:54.683 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:18:54.683 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:18:54.683 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-03-26 02:18:29.268015806 +0000 2024-03-26T02:18:54.683 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-03-26 02:16:04.385735990 +0000 2024-03-26T02:18:54.683 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-03-26 02:16:04.385735990 +0000 2024-03-26T02:18:54.683 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-03-26 02:16:04.385735990 +0000 2024-03-26T02:18:54.683 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T02:18:54.748 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-03-26T02:18:54.748 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-03-26T02:18:54.748 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000182804 s, 2.8 MB/s 2024-03-26T02:18:54.749 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T02:18:54.805 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-26T02:18:54.805 DEBUG:teuthology.orchestra.run.smithi172:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T02:18:54.820 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T02:18:54.821 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_1 2024-03-26T02:18:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T02:18:54.876 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:18:54.876 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-26T02:18:54.876 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:18:54.876 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:18:54.876 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-26 02:18:38.569303714 +0000 2024-03-26T02:18:54.876 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-26 02:16:03.338654152 +0000 2024-03-26T02:18:54.877 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-26 02:16:03.338654152 +0000 2024-03-26T02:18:54.877 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-26 02:16:03.338654152 +0000 2024-03-26T02:18:54.877 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T02:18:54.943 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-26T02:18:54.943 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-26T02:18:54.943 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000262394 s, 2.0 MB/s 2024-03-26T02:18:54.945 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T02:18:55.000 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_2 2024-03-26T02:18:55.055 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T02:18:55.055 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:18:55.055 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T02:18:55.055 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:18:55.055 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:18:55.055 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-26 02:18:38.570303688 +0000 2024-03-26T02:18:55.056 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-26 02:16:03.677645598 +0000 2024-03-26T02:18:55.056 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-26 02:16:03.677645598 +0000 2024-03-26T02:18:55.056 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-26 02:16:03.677645598 +0000 2024-03-26T02:18:55.056 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T02:18:55.122 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-26T02:18:55.122 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-26T02:18:55.122 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000212699 s, 2.4 MB/s 2024-03-26T02:18:55.123 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T02:18:55.179 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_3 2024-03-26T02:18:55.233 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T02:18:55.233 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:18:55.233 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T02:18:55.233 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:18:55.233 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:18:55.234 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-26 02:18:38.570303688 +0000 2024-03-26T02:18:55.234 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-26 02:16:04.009637221 +0000 2024-03-26T02:18:55.234 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-26 02:16:04.009637221 +0000 2024-03-26T02:18:55.234 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-26 02:16:04.009637221 +0000 2024-03-26T02:18:55.234 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T02:18:55.299 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-26T02:18:55.299 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-26T02:18:55.299 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000224312 s, 2.3 MB/s 2024-03-26T02:18:55.300 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T02:18:55.357 DEBUG:teuthology.orchestra.run.smithi172:> stat /dev/vg_nvme/lv_4 2024-03-26T02:18:55.411 INFO:teuthology.orchestra.run.smithi172.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T02:18:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T02:18:55.412 INFO:teuthology.orchestra.run.smithi172.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-26T02:18:55.412 INFO:teuthology.orchestra.run.smithi172.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T02:18:55.412 INFO:teuthology.orchestra.run.smithi172.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T02:18:55.412 INFO:teuthology.orchestra.run.smithi172.stdout:Access: 2024-03-26 02:18:38.571303664 +0000 2024-03-26T02:18:55.412 INFO:teuthology.orchestra.run.smithi172.stdout:Modify: 2024-03-26 02:16:04.362628314 +0000 2024-03-26T02:18:55.412 INFO:teuthology.orchestra.run.smithi172.stdout:Change: 2024-03-26 02:16:04.362628314 +0000 2024-03-26T02:18:55.412 INFO:teuthology.orchestra.run.smithi172.stdout: Birth: 2024-03-26 02:16:04.362628314 +0000 2024-03-26T02:18:55.412 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T02:18:55.499 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records in 2024-03-26T02:18:55.499 INFO:teuthology.orchestra.run.smithi172.stderr:1+0 records out 2024-03-26T02:18:55.499 INFO:teuthology.orchestra.run.smithi172.stderr:512 bytes copied, 0.000176637 s, 2.9 MB/s 2024-03-26T02:18:55.501 DEBUG:teuthology.orchestra.run.smithi172:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T02:18:55.517 INFO:tasks.cephadm:Deploying osd.0 on smithi164 with /dev/vg_nvme/lv_4... 2024-03-26T02:18:55.517 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T02:18:55.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:55 smithi172 ceph-mon[25688]: Deploying daemon mgr.x on smithi172 2024-03-26T02:18:55.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:55 smithi164 ceph-mon[24851]: Deploying daemon mgr.x on smithi172 2024-03-26T02:18:55.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:55 smithi164 ceph-mon[29512]: Deploying daemon mgr.x on smithi172 2024-03-26T02:18:56.721 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:56 smithi172 ceph-mon[25688]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:56.721 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:56 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:56.721 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:56 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:56.721 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:56 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:56.721 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:56 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:56.722 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:56 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:56.627+0000 7fb5b69dc000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:18:56.722 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:56 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:56.720+0000 7fb5b69dc000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:18:56.729 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[29512]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:56.729 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:56.729 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:56.729 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:56.730 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:56.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[24851]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:56.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:56.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:56.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:56.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:56 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T02:18:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T02:18:57.462 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-26T02:18:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-26T02:18:57.462 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113815 s, 921 MB/s 2024-03-26T02:18:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-26T02:18:57.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:57.339+0000 7fb5b69dc000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:18:57.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:57.428+0000 7fb5b69dc000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:18:57.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:57.510+0000 7fb5b69dc000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:18:57.932 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:57.660+0000 7fb5b69dc000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:18:57.933 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:57.740+0000 7fb5b69dc000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:18:57.933 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:57.849+0000 7fb5b69dc000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:18:57.933 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:57.931+0000 7fb5b69dc000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:18:58.061 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_4 2024-03-26T02:18:58.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:58.094+0000 7fb5b69dc000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:18:58.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:58.179+0000 7fb5b69dc000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:18:58.422 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:58.422 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:58.422 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[24851]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:18:58.422 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:18:58.422 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:18:58.422 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:58.422 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[24851]: Reconfiguring daemon mgr.y on smithi164 2024-03-26T02:18:58.422 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[24851]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:58.423 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:58.423 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:58.423 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[29512]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:18:58.423 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:18:58.423 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:18:58.423 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:58.423 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[29512]: Reconfiguring daemon mgr.y on smithi164 2024-03-26T02:18:58.423 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:58 smithi164 ceph-mon[29512]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:58.603 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:58.603 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:58.603 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-mon[25688]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T02:18:58.603 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:18:58.603 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:18:58.604 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:58.604 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-mon[25688]: Reconfiguring daemon mgr.y on smithi164 2024-03-26T02:18:58.604 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:58 smithi172 ceph-mon[25688]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:18:59.532 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:59 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:18:59.533 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:59 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:18:59.533 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:18:59 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:59.533 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:59.215+0000 7fb5b69dc000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:18:59.533 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:59.342+0000 7fb5b69dc000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:18:59.533 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:59.531+0000 7fb5b69dc000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:18:59.614 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:59 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:18:59.614 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:59 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:18:59.614 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:18:59 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:59.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:59 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:18:59.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:59 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:18:59.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:18:59 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:18:59.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:59.616+0000 7fb5b69dc000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:18:59.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:18:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:18:59.748+0000 7fb5b69dc000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:19:00.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:19:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:19:00.062+0000 7fb5b69dc000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:19:00.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:19:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:19:00.277+0000 7fb5b69dc000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:19:00.360 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:19:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:19:00.359+0000 7fb5b69dc000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:19:00.676 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:00 smithi164 ceph-mon[24851]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:00.676 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:00.677 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:00.677 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:00.677 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:00 smithi164 ceph-mon[29512]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:00.677 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:00.677 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:00.677 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:00.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:19:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:19:00.526+0000 7fb5b69dc000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:19:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:00 smithi172 ceph-mon[25688]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[29512]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[24851]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:01.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:01.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:01.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:01.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:01 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:01.717 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:19:01.426+0000 7fb5b69dc000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:19:01.717 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:19:01.716+0000 7fb5b69dc000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:19:01.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-mon[25688]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:01.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:01.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:01.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:01.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:01.718 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:02.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:19:01.809+0000 7fb5b69dc000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:19:02.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:19:01 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:19:01.954+0000 7fb5b69dc000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:19:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[24851]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[24851]: Standby manager daemon x started 2024-03-26T02:19:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:19:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:19:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:19:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:19:02.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[29512]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:02.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[29512]: Standby manager daemon x started 2024-03-26T02:19:02.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:19:02.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:19:02.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:19:02.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:02 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:19:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:02 smithi172 ceph-mon[25688]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:02.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:02 smithi172 ceph-mon[25688]: Standby manager daemon x started 2024-03-26T02:19:02.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:02 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:19:02.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:02 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:19:02.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:02 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:19:02.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:02 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/2703404663' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:19:03.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[24851]: mgrmap e14: y(active, since 63s), standbys: x 2024-03-26T02:19:03.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2135887176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa926af6-17d0-4176-b716-b668e463738d"}]: dispatch 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2135887176' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa926af6-17d0-4176-b716-b668e463738d"}]': finished 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[24851]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1093572844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[29512]: mgrmap e14: y(active, since 63s), standbys: x 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2135887176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa926af6-17d0-4176-b716-b668e463738d"}]: dispatch 2024-03-26T02:19:03.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2135887176' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa926af6-17d0-4176-b716-b668e463738d"}]': finished 2024-03-26T02:19:03.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[29512]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:19:03.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:03.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:03 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1093572844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:03 smithi172 ceph-mon[25688]: mgrmap e14: y(active, since 63s), standbys: x 2024-03-26T02:19:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:03 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:19:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:03 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2135887176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fa926af6-17d0-4176-b716-b668e463738d"}]: dispatch 2024-03-26T02:19:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:03 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2135887176' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fa926af6-17d0-4176-b716-b668e463738d"}]': finished 2024-03-26T02:19:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:03 smithi172 ceph-mon[25688]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T02:19:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:03 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:03 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1093572844' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:04 smithi164 ceph-mon[24851]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:04 smithi164 ceph-mon[29512]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:04 smithi172 ceph-mon[25688]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:06.776 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:06 smithi164 ceph-mon[24851]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:06.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:06 smithi164 ceph-mon[29512]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:06 smithi172 ceph-mon[25688]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:08.678 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:08 smithi164 ceph-mon[24851]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:08.678 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:08 smithi164 ceph-mon[29512]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:08.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:08 smithi172 ceph-mon[25688]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:09.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:09 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:19:09.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:09 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:09 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:19:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:09 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:09 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:19:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:09 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:10.676 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:10 smithi164 ceph-mon[24851]: Deploying daemon osd.0 on smithi164 2024-03-26T02:19:10.676 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:10 smithi164 ceph-mon[24851]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:10.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:10 smithi164 ceph-mon[29512]: Deploying daemon osd.0 on smithi164 2024-03-26T02:19:10.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:10 smithi164 ceph-mon[29512]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:10.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:10 smithi172 ceph-mon[25688]: Deploying daemon osd.0 on smithi164 2024-03-26T02:19:10.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:10 smithi172 ceph-mon[25688]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:11.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:11 smithi164 ceph-mon[24851]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:11.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:11 smithi164 ceph-mon[29512]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:11.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:11 smithi172 ceph-mon[25688]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:14.454 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[29512]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:14.454 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:14.454 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:14.454 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:14.454 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:14.454 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[24851]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:14.454 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:14.454 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:14.455 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:14.455 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:14 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:14.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:14 smithi172 ceph-mon[25688]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:14 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:14 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:14 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:14 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:16.506 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:16 smithi164 ceph-mon[24851]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:16.506 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:16 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:16.506 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:16 smithi164 ceph-mon[29512]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:16.506 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:16 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:16 smithi172 ceph-mon[25688]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:16 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:17.097 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 0 on host 'smithi164' 2024-03-26T02:19:17.729 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:17.729 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[29512]: from='osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[29512]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[29512]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[24851]: from='osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[24851]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:17.730 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:17.731 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:17.731 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:17.731 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:17 smithi164 ceph-mon[24851]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:17.751 DEBUG:teuthology.orchestra.run.smithi164:osd.0> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T02:19:17.753 INFO:tasks.cephadm:Deploying osd.1 on smithi164 with /dev/vg_nvme/lv_3... 2024-03-26T02:19:17.754 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T02:19:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:17 smithi172 ceph-mon[25688]: from='osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:19:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:17 smithi172 ceph-mon[25688]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:19:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:17 smithi172 ceph-mon[25688]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:18.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[24851]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:19:18.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[24851]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:19:18.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:18.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[24851]: from='osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:18.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[24851]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[29512]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:19:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[29512]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:19:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[29512]: from='osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-mon[29512]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:18.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[32388]: 2024-03-26T02:19:18.533+0000 7fbccb84a700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T02:19:18.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:19:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[32388]: 2024-03-26T02:19:18.535+0000 7fbcc51df700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:19:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:18 smithi172 ceph-mon[25688]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:19:18.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:18 smithi172 ceph-mon[25688]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T02:19:18.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:18 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:18.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:18 smithi172 ceph-mon[25688]: from='osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:18.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:18 smithi172 ceph-mon[25688]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:19.589 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[24851]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:19:19.589 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[24851]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:19:19.589 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:19.589 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:19.589 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[24851]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:19.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[29512]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:19:19.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[29512]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:19:19.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:19.590 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:19.591 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:19 smithi164 ceph-mon[29512]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:19 smithi172 ceph-mon[25688]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:19:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:19 smithi172 ceph-mon[25688]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T02:19:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:19 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:19 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:19 smithi172 ceph-mon[25688]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T02:19:20.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[24851]: purged_snaps scrub starts 2024-03-26T02:19:20.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[24851]: purged_snaps scrub ok 2024-03-26T02:19:20.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[24851]: osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558] boot 2024-03-26T02:19:20.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[24851]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:19:20.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:20.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[29512]: purged_snaps scrub starts 2024-03-26T02:19:20.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[29512]: purged_snaps scrub ok 2024-03-26T02:19:20.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[29512]: osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558] boot 2024-03-26T02:19:20.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[29512]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:19:20.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:20 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:20.604 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T02:19:20.604 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T02:19:20.604 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-26T02:19:20.604 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-26T02:19:20.604 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0152292 s, 689 MB/s 2024-03-26T02:19:20.604 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-26T02:19:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:20 smithi172 ceph-mon[25688]: purged_snaps scrub starts 2024-03-26T02:19:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:20 smithi172 ceph-mon[25688]: purged_snaps scrub ok 2024-03-26T02:19:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:20 smithi172 ceph-mon[25688]: osd.0 [v2:172.21.15.164:6802/378049558,v1:172.21.15.164:6803/378049558] boot 2024-03-26T02:19:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:20 smithi172 ceph-mon[25688]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T02:19:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:20 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:19:21.484 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_3 2024-03-26T02:19:21.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:21 smithi164 ceph-mon[24851]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:19:21.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:21 smithi164 ceph-mon[24851]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:21.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:21 smithi164 ceph-mon[29512]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:19:21.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:21 smithi164 ceph-mon[29512]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:21 smithi172 ceph-mon[25688]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T02:19:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:21 smithi172 ceph-mon[25688]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:23.424 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi164 2024-03-26T02:19:23.424 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:23.424 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:23.424 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi164 to 16064M 2024-03-26T02:19:23.424 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:23.424 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:23.424 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi164 2024-03-26T02:19:23.424 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:23.425 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:23.425 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi164 to 16064M 2024-03-26T02:19:23.425 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:23.425 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:23 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:23 smithi172 ceph-mon[25688]: Detected new or changed devices on smithi164 2024-03-26T02:19:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:23 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:23 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:23 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi164 to 16064M 2024-03-26T02:19:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:23 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:23 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:24.542 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:24 smithi164 ceph-mon[24851]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:24.543 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:24 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:24.543 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:24 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:24.543 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:24 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:24.543 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:24 smithi164 ceph-mon[29512]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:24.543 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:24 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:24.543 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:24 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:24.543 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:24 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:24.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:24 smithi172 ceph-mon[25688]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:24.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:24 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:24 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:24 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:25.464 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:25 smithi164 ceph-mon[24851]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:25.464 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:25 smithi164 ceph-mon[29512]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:25.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:25 smithi172 ceph-mon[25688]: from='client.24145 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:26.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[24851]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:26.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1311089503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "662f7218-b733-4bd5-9c05-305d11ec6b26"}]: dispatch 2024-03-26T02:19:26.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1311089503' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "662f7218-b733-4bd5-9c05-305d11ec6b26"}]': finished 2024-03-26T02:19:26.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[24851]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:19:26.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:26.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[29512]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:26.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1311089503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "662f7218-b733-4bd5-9c05-305d11ec6b26"}]: dispatch 2024-03-26T02:19:26.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1311089503' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "662f7218-b733-4bd5-9c05-305d11ec6b26"}]': finished 2024-03-26T02:19:26.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[29512]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:19:26.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:26 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:26.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:26 smithi172 ceph-mon[25688]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:26.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:26 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1311089503' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "662f7218-b733-4bd5-9c05-305d11ec6b26"}]: dispatch 2024-03-26T02:19:26.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:26 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1311089503' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "662f7218-b733-4bd5-9c05-305d11ec6b26"}]': finished 2024-03-26T02:19:26.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:26 smithi172 ceph-mon[25688]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T02:19:26.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:26 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:27.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:27 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1201023552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:27.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:27 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1201023552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:27.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:27 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1201023552' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:28.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:28 smithi164 ceph-mon[24851]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:28.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:28 smithi164 ceph-mon[29512]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:28.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:28 smithi172 ceph-mon[25688]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:30.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:30 smithi164 ceph-mon[24851]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:30.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:30 smithi164 ceph-mon[29512]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:30 smithi172 ceph-mon[25688]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:32.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:32 smithi164 ceph-mon[24851]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:32.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:32 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:19:32.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:32 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:32.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:32 smithi164 ceph-mon[29512]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:32.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:32 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:19:32.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:32 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:32 smithi172 ceph-mon[25688]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:32.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:32 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:19:32.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:32 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:33.563 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:33 smithi164 ceph-mon[24851]: Deploying daemon osd.1 on smithi164 2024-03-26T02:19:33.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:33 smithi164 ceph-mon[29512]: Deploying daemon osd.1 on smithi164 2024-03-26T02:19:33.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:33 smithi172 ceph-mon[25688]: Deploying daemon osd.1 on smithi164 2024-03-26T02:19:34.511 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:34 smithi164 ceph-mon[29512]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:34.511 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:34 smithi164 ceph-mon[24851]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:34 smithi172 ceph-mon[25688]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:36.770 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:36 smithi164 ceph-mon[24851]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:36.771 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:36 smithi164 ceph-mon[29512]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:36 smithi172 ceph-mon[25688]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:38.628 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:38.628 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[24851]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:38.628 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:38.628 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:38.628 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:38.629 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:38.629 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[29512]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:38.629 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:38.629 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:38.629 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:38 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:38.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:38 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:38.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:38 smithi172 ceph-mon[25688]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:38.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:38 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:38.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:38 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:38.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:38 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:40.410 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 1 on host 'smithi164' 2024-03-26T02:19:40.656 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:40 smithi164 ceph-mon[24851]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:40.657 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:40 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:40.657 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:40 smithi164 ceph-mon[29512]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:40.657 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:40 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:40 smithi172 ceph-mon[25688]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:40 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:40.981 DEBUG:teuthology.orchestra.run.smithi164:osd.1> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T02:19:40.983 INFO:tasks.cephadm:Deploying osd.2 on smithi164 with /dev/vg_nvme/lv_2... 2024-03-26T02:19:40.984 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T02:19:41.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:41 smithi164 ceph-mon[24851]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:19:41.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:41 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:41.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:41 smithi164 ceph-mon[29512]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:19:41.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:41 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:41.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:41 smithi172 ceph-mon[25688]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:19:41.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:41 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:42.659 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[24851]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:42.659 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[24851]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:19:42.659 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[24851]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:19:42.659 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:42.659 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[24851]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:42.660 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[29512]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:42.660 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[29512]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:19:42.660 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[29512]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:19:42.660 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:42.660 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-mon[29512]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:42.660 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[35369]: 2024-03-26T02:19:42.437+0000 7fb7a00e3700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T02:19:42.660 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:19:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[35369]: 2024-03-26T02:19:42.441+0000 7fb79aa7a700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:19:42.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:42 smithi172 ceph-mon[25688]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:42.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:42 smithi172 ceph-mon[25688]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:19:42.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:42 smithi172 ceph-mon[25688]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T02:19:42.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:42.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:42 smithi172 ceph-mon[25688]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:19:43.679 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T02:19:43.679 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T02:19:43.679 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-26T02:19:43.679 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-26T02:19:43.680 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122138 s, 859 MB/s 2024-03-26T02:19:43.680 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-26T02:19:43.701 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:43 smithi164 ceph-mon[24851]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:19:43.701 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:43 smithi164 ceph-mon[24851]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:19:43.702 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:43 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:43.702 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:43 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:43.702 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:43 smithi164 ceph-mon[29512]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:19:43.702 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:43 smithi164 ceph-mon[29512]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:19:43.702 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:43 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:43.702 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:43 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:43.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:43 smithi172 ceph-mon[25688]: from='osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:19:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:43 smithi172 ceph-mon[25688]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T02:19:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:43 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:43 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:44.326 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_2 2024-03-26T02:19:44.630 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[29512]: purged_snaps scrub starts 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[29512]: purged_snaps scrub ok 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[29512]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[29512]: osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680] boot 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[29512]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[24851]: purged_snaps scrub starts 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[24851]: purged_snaps scrub ok 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[24851]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:44.631 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:44.632 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[24851]: osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680] boot 2024-03-26T02:19:44.632 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[24851]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:19:44.632 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:44 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:44 smithi172 ceph-mon[25688]: purged_snaps scrub starts 2024-03-26T02:19:44.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:44 smithi172 ceph-mon[25688]: purged_snaps scrub ok 2024-03-26T02:19:44.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:44 smithi172 ceph-mon[25688]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-03-26T02:19:44.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:44 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:44.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:44 smithi172 ceph-mon[25688]: osd.1 [v2:172.21.15.164:6810/3337400680,v1:172.21.15.164:6811/3337400680] boot 2024-03-26T02:19:44.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:44 smithi172 ceph-mon[25688]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T02:19:44.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:44 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:19:46.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:46 smithi164 ceph-mon[24851]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:19:46.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:46 smithi164 ceph-mon[24851]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:46.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:46 smithi164 ceph-mon[29512]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:19:46.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:46 smithi164 ceph-mon[29512]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:46.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:46 smithi172 ceph-mon[25688]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T02:19:46.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:46 smithi172 ceph-mon[25688]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:47.571 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi164 2024-03-26T02:19:47.571 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.571 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi164 to 8032M 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:47.572 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi164 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi164 to 8032M 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:47.573 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:47.574 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: Detected new or changed devices on smithi164 2024-03-26T02:19:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:19:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi164 to 8032M 2024-03-26T02:19:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:47.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:47.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:47.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:19:47.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:19:47.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:19:47.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:48.514 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:48 smithi164 ceph-mon[24851]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:48.515 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:48 smithi164 ceph-mon[24851]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:48.515 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:48 smithi164 ceph-mon[29512]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:48.515 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:48 smithi164 ceph-mon[29512]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:48 smithi172 ceph-mon[25688]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:19:48.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:48 smithi172 ceph-mon[25688]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3936088708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]: dispatch 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]: dispatch 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]': finished 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[24851]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3936088708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]: dispatch 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]: dispatch 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]': finished 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[29512]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:19:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:49 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:19:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:49 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3936088708' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]: dispatch 2024-03-26T02:19:49.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:49 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]: dispatch 2024-03-26T02:19:49.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:49 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cea9695c-f306-468a-8071-debc4326b99b"}]': finished 2024-03-26T02:19:49.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:49 smithi172 ceph-mon[25688]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T02:19:49.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:49 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:19:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:50 smithi164 ceph-mon[24851]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:50 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/4157416886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:50.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:50 smithi164 ceph-mon[29512]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:50 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/4157416886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:50 smithi172 ceph-mon[25688]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:50 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/4157416886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:19:52.643 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:52 smithi164 ceph-mon[24851]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:52.643 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:52 smithi164 ceph-mon[29512]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:52 smithi172 ceph-mon[25688]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:54.259 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:54 smithi164 ceph-mon[24851]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:54.259 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:54 smithi164 ceph-mon[29512]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:54 smithi172 ceph-mon[25688]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:55.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:55 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:19:55.599 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:55 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:55.600 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:55 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:19:55.600 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:55 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:55.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:55 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:19:55.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:55 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:56.692 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:56 smithi164 ceph-mon[24851]: Deploying daemon osd.2 on smithi164 2024-03-26T02:19:56.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:56 smithi164 ceph-mon[24851]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:56.693 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:56 smithi164 ceph-mon[29512]: Deploying daemon osd.2 on smithi164 2024-03-26T02:19:56.693 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:56 smithi164 ceph-mon[29512]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:56 smithi172 ceph-mon[25688]: Deploying daemon osd.2 on smithi164 2024-03-26T02:19:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:56 smithi172 ceph-mon[25688]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:58.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:58 smithi164 ceph-mon[24851]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:58.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:58 smithi164 ceph-mon[29512]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:58.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:58 smithi172 ceph-mon[25688]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:19:59.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:59 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:19:59.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:19:59 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:19:59.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:59 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:19:59.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:19:59 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:19:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:59 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:19:59.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:19:59 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:20:00.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[24851]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:00.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[24851]: overall HEALTH_OK 2024-03-26T02:20:00.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:00.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:00.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:00.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:00.751 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[29512]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:00.752 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[29512]: overall HEALTH_OK 2024-03-26T02:20:00.752 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:00.752 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:00.752 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:00.752 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:00 smithi172 ceph-mon[25688]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:00 smithi172 ceph-mon[25688]: overall HEALTH_OK 2024-03-26T02:20:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:02.638 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:02 smithi164 ceph-mon[24851]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:02.638 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:02 smithi164 ceph-mon[29512]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:02.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:02 smithi172 ceph-mon[25688]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:03.459 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 2 on host 'smithi164' 2024-03-26T02:20:04.078 DEBUG:teuthology.orchestra.run.smithi164:osd.2> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T02:20:04.081 INFO:tasks.cephadm:Deploying osd.3 on smithi164 with /dev/vg_nvme/lv_1... 2024-03-26T02:20:04.081 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T02:20:04.606 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[24851]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:04.606 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:04.606 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[24851]: from='osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:20:04.606 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[24851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:20:04.606 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:04.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[29512]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:04.606 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:04.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[29512]: from='osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:20:04.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[29512]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:20:04.607 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:04 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:04 smithi172 ceph-mon[25688]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:04 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:04 smithi172 ceph-mon[25688]: from='osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:20:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:04 smithi172 ceph-mon[25688]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:20:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:04 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:05.722 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[24851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:20:05.722 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[24851]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:20:05.722 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:05.722 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[24851]: from='osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:05.722 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[24851]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[29512]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:20:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[29512]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:20:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[29512]: from='osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:05.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-mon[29512]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:05.723 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[38296]: 2024-03-26T02:20:05.470+0000 7f811aa2a700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T02:20:05.723 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:20:05 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[38296]: 2024-03-26T02:20:05.473+0000 7f8114bc0700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:20:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:05 smithi172 ceph-mon[25688]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:20:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:05 smithi172 ceph-mon[25688]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T02:20:05.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:05.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:05 smithi172 ceph-mon[25688]: from='osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:05.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:05 smithi172 ceph-mon[25688]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:06.443 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T02:20:06.443 INFO:teuthology.orchestra.run.smithi164.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T02:20:06.443 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10+0 records in 2024-03-26T02:20:06.443 INFO:teuthology.orchestra.run.smithi164.stderr:10+0 records out 2024-03-26T02:20:06.443 INFO:teuthology.orchestra.run.smithi164.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124474 s, 842 MB/s 2024-03-26T02:20:06.443 INFO:teuthology.orchestra.run.smithi164.stderr:--> Zapping successful for: 2024-03-26T02:20:06.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[24851]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:06.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[24851]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:20:06.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[24851]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:20:06.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:06.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[29512]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[29512]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:20:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[29512]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:20:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:06 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:06 smithi172 ceph-mon[25688]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:06 smithi172 ceph-mon[25688]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:20:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:06 smithi172 ceph-mon[25688]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T02:20:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:06 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:06 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:07.011 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi164:vg_nvme/lv_1 2024-03-26T02:20:07.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[29512]: purged_snaps scrub starts 2024-03-26T02:20:07.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[29512]: purged_snaps scrub ok 2024-03-26T02:20:07.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[29512]: osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553] boot 2024-03-26T02:20:07.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[29512]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:20:07.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:07.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-26T02:20:07.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[24851]: purged_snaps scrub starts 2024-03-26T02:20:07.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[24851]: purged_snaps scrub ok 2024-03-26T02:20:07.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[24851]: osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553] boot 2024-03-26T02:20:07.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[24851]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:20:07.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:07.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:07 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-26T02:20:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:07 smithi172 ceph-mon[25688]: purged_snaps scrub starts 2024-03-26T02:20:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:07 smithi172 ceph-mon[25688]: purged_snaps scrub ok 2024-03-26T02:20:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:07 smithi172 ceph-mon[25688]: osd.2 [v2:172.21.15.164:6818/851694553,v1:172.21.15.164:6819/851694553] boot 2024-03-26T02:20:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:07 smithi172 ceph-mon[25688]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T02:20:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:07 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:20:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:07 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-26T02:20:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:08 smithi164 ceph-mon[24851]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:08 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-26T02:20:08.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:08 smithi164 ceph-mon[24851]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:20:08.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:08 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:20:08.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:08 smithi164 ceph-mon[29512]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:08.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:08 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-26T02:20:08.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:08 smithi164 ceph-mon[29512]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:20:08.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:08 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:20:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:08 smithi172 ceph-mon[25688]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-03-26T02:20:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:08 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-26T02:20:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:08 smithi172 ceph-mon[25688]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T02:20:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:08 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:20:09.425 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39601]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:20:09.425 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39601]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:09.425 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39601]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:09.695 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39616]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:20:09.695 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39616]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:09.696 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39616]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:09.967 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39622]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:20:09.968 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39622]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:09.968 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39622]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:09.968 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:09 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:20:09.968 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:09 smithi164 ceph-mon[24851]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:20:09.968 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:09 smithi164 ceph-mon[24851]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:09.968 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:09 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:09.968 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:09 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:20:09.969 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:09 smithi164 ceph-mon[29512]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:20:09.969 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:09 smithi164 ceph-mon[29512]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:09.969 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:09 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:09 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:20:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:09 smithi172 ceph-mon[25688]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T02:20:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:09 smithi172 ceph-mon[25688]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:09 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:10.240 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39664]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:20:10.240 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39664]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:10.240 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:20:09 smithi164 sudo[39664]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:10.492 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39726]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:20:10.492 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39726]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:10.492 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39726]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:10.753 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39752]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:20:10.754 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39752]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:10.754 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39735]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:20:10.754 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39735]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:10.754 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39735]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:11.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39752]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:11.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39758]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:20:11.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:10 smithi164 sudo[39758]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:11.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 sudo[39758]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:11.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:20:11.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:11.265 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:11.265 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:11.265 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:11.265 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi164 2024-03-26T02:20:11.265 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.265 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.265 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.265 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi164 to 5354M 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:11.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi164 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi164 to 5354M 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 sudo[39761]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 sudo[39761]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:11.267 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 sudo[39761]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:11.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi164:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: Detected new or changed devices on smithi164 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi164 to 5354M 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:11.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:11.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:11.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:11.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 sudo[39784]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:20:11.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 sudo[39784]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:11.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:11 smithi164 sudo[39784]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:12.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 sudo[27210]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T02:20:12.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 sudo[27210]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:12.005 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:11 smithi172 sudo[27210]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: pgmap v77: 1 pgs: 1 active+clean; 513 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T02:20:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: pgmap v77: 1 pgs: 1 active+clean; 513 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:12.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:12.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 sudo[27213]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 sudo[27213]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 sudo[27213]: pam_unix(sudo:session): session closed for user root 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: pgmap v77: 1 pgs: 1 active+clean; 513 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:12.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:20:12.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:20:12.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:20:13.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[24851]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[24851]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/531884440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b90e1e0c-0eee-46e3-b56c-54e8aea3bb75"}]: dispatch 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/531884440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b90e1e0c-0eee-46e3-b56c-54e8aea3bb75"}]': finished 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[24851]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3009876184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[29512]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:13.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/531884440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b90e1e0c-0eee-46e3-b56c-54e8aea3bb75"}]: dispatch 2024-03-26T02:20:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/531884440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b90e1e0c-0eee-46e3-b56c-54e8aea3bb75"}]': finished 2024-03-26T02:20:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[29512]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T02:20:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:13 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3009876184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:20:13.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:13 smithi172 ceph-mon[25688]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T02:20:13.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:13 smithi172 ceph-mon[25688]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T02:20:13.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:13 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:13 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/531884440' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b90e1e0c-0eee-46e3-b56c-54e8aea3bb75"}]: dispatch 2024-03-26T02:20:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:13 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/531884440' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b90e1e0c-0eee-46e3-b56c-54e8aea3bb75"}]': finished 2024-03-26T02:20:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:13 smithi172 ceph-mon[25688]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T02:20:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:13 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:13 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3009876184' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:20:14.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:14 smithi164 ceph-mon[24851]: pgmap v79: 1 pgs: 1 active+clean; 513 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:14.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:14 smithi164 ceph-mon[29512]: pgmap v79: 1 pgs: 1 active+clean; 513 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:14.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:14 smithi172 ceph-mon[25688]: pgmap v79: 1 pgs: 1 active+clean; 513 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:16.525 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:16 smithi164 ceph-mon[24851]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 195 KiB/s wr, 12 op/s 2024-03-26T02:20:16.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:16 smithi164 ceph-mon[29512]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 195 KiB/s wr, 12 op/s 2024-03-26T02:20:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:16 smithi172 ceph-mon[25688]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 195 KiB/s wr, 12 op/s 2024-03-26T02:20:18.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:18 smithi164 ceph-mon[24851]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 144 KiB/s wr, 8 op/s 2024-03-26T02:20:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:18 smithi164 ceph-mon[29512]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 144 KiB/s wr, 8 op/s 2024-03-26T02:20:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:18 smithi172 ceph-mon[25688]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.6 KiB/s rd, 144 KiB/s wr, 8 op/s 2024-03-26T02:20:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:20:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:19 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:19 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:20:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:19 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:19 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:20:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:19 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:20 smithi164 ceph-mon[24851]: Deploying daemon osd.3 on smithi164 2024-03-26T02:20:20.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:20 smithi164 ceph-mon[24851]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-03-26T02:20:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:20 smithi164 ceph-mon[29512]: Deploying daemon osd.3 on smithi164 2024-03-26T02:20:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:20 smithi164 ceph-mon[29512]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-03-26T02:20:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:20 smithi172 ceph-mon[25688]: Deploying daemon osd.3 on smithi164 2024-03-26T02:20:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:20 smithi172 ceph-mon[25688]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.1 KiB/s rd, 114 KiB/s wr, 7 op/s 2024-03-26T02:20:22.747 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:22 smithi164 ceph-mon[24851]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 111 KiB/s wr, 6 op/s 2024-03-26T02:20:22.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:22 smithi164 ceph-mon[29512]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 111 KiB/s wr, 6 op/s 2024-03-26T02:20:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:22 smithi172 ceph-mon[25688]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.0 KiB/s rd, 111 KiB/s wr, 6 op/s 2024-03-26T02:20:24.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[24851]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 100 KiB/s wr, 6 op/s 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[29512]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 100 KiB/s wr, 6 op/s 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:24 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:24.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:24 smithi172 ceph-mon[25688]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 100 KiB/s wr, 6 op/s 2024-03-26T02:20:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:24 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:24 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:24 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:24 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:26.665 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:26 smithi164 ceph-mon[24851]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 92 KiB/s wr, 5 op/s 2024-03-26T02:20:26.665 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:26 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:26.665 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:26 smithi164 ceph-mon[29512]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 92 KiB/s wr, 5 op/s 2024-03-26T02:20:26.665 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:26 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:26 smithi172 ceph-mon[25688]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 92 KiB/s wr, 5 op/s 2024-03-26T02:20:26.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:26 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:27.109 INFO:teuthology.orchestra.run.smithi164.stdout:Created osd(s) 3 on host 'smithi164' 2024-03-26T02:20:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[24851]: from='osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:20:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[24851]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:20:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:27.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[29512]: from='osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:20:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[29512]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:20:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:27 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:27.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:27 smithi172 ceph-mon[25688]: from='osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:20:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:27 smithi172 ceph-mon[25688]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:20:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:27 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:27 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:27 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:27 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:27 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:27.827 DEBUG:teuthology.orchestra.run.smithi164:osd.3> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T02:20:27.829 INFO:tasks.cephadm:Deploying osd.4 on smithi172 with /dev/vg_nvme/lv_4... 2024-03-26T02:20:27.829 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T02:20:28.778 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[24851]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[24851]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[24851]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[24851]: from='osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[24851]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[29512]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[29512]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[29512]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:28.779 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[29512]: from='osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:28.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-mon[29512]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:28.780 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[41283]: 2024-03-26T02:20:28.450+0000 7f6f10c3f700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T02:20:28.780 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:20:28 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[41283]: 2024-03-26T02:20:28.454+0000 7f6f0b5d6700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:20:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:28 smithi172 ceph-mon[25688]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:28 smithi172 ceph-mon[25688]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:20:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:28 smithi172 ceph-mon[25688]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T02:20:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:28 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:28 smithi172 ceph-mon[25688]: from='osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:28 smithi172 ceph-mon[25688]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:20:29.657 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T02:20:29.658 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T02:20:29.658 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-26T02:20:29.658 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-26T02:20:29.658 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113504 s, 924 MB/s 2024-03-26T02:20:29.658 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-26T02:20:29.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:29 smithi164 ceph-mon[24851]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:20:29.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:29 smithi164 ceph-mon[24851]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T02:20:29.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:29 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:29.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:29 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:29.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:29 smithi164 ceph-mon[29512]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:20:29.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:29 smithi164 ceph-mon[29512]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T02:20:29.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:29 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:29.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:29 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:29 smithi172 ceph-mon[25688]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]': finished 2024-03-26T02:20:29.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:29 smithi172 ceph-mon[25688]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T02:20:29.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:29 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:29.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:29 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:30.262 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_4 2024-03-26T02:20:30.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[29512]: purged_snaps scrub starts 2024-03-26T02:20:30.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[29512]: purged_snaps scrub ok 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[29512]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[29512]: osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232] boot 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[29512]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[24851]: purged_snaps scrub starts 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[24851]: purged_snaps scrub ok 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[24851]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[24851]: osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232] boot 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[24851]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T02:20:30.744 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:30 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:30 smithi172 ceph-mon[25688]: purged_snaps scrub starts 2024-03-26T02:20:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:30 smithi172 ceph-mon[25688]: purged_snaps scrub ok 2024-03-26T02:20:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:30 smithi172 ceph-mon[25688]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-03-26T02:20:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:30 smithi172 ceph-mon[25688]: osd.3 [v2:172.21.15.164:6826/3222350232,v1:172.21.15.164:6827/3222350232] boot 2024-03-26T02:20:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:30 smithi172 ceph-mon[25688]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T02:20:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:30 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:20:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:31 smithi172 ceph-mon[25688]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T02:20:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:31 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:31 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:31 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:31 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:31 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:31 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:31 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:31.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[24851]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T02:20:31.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[29512]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:31.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:31.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:31 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:32.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:32 smithi172 ceph-mon[25688]: Detected new or changed devices on smithi164 2024-03-26T02:20:32.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:32 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi164 to 4016M 2024-03-26T02:20:32.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:32 smithi172 ceph-mon[25688]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:32.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:32 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:32.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:32 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:32.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:32 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:32.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi164 2024-03-26T02:20:32.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi164 to 4016M 2024-03-26T02:20:32.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[29512]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:32.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:32.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:32.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:32.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi164 2024-03-26T02:20:32.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi164 to 4016M 2024-03-26T02:20:32.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[24851]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:32.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:32.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:32.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:32 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:33.565 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:33 smithi172 ceph-mon[25688]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:33.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:33 smithi164 ceph-mon[24851]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:33.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:33 smithi164 ceph-mon[29512]: from='client.34205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[24851]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/843762967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]: dispatch 2024-03-26T02:20:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]: dispatch 2024-03-26T02:20:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]': finished 2024-03-26T02:20:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[24851]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T02:20:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[29512]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/843762967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]: dispatch 2024-03-26T02:20:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]: dispatch 2024-03-26T02:20:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]': finished 2024-03-26T02:20:34.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[29512]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T02:20:34.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:34 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:34.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:34 smithi172 ceph-mon[25688]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:34.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:34 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/843762967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]: dispatch 2024-03-26T02:20:34.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:34 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]: dispatch 2024-03-26T02:20:34.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:34 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f73412ca-7838-48e2-a5dc-897dccf6b4bf"}]': finished 2024-03-26T02:20:34.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:34 smithi172 ceph-mon[25688]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T02:20:34.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:34 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:35.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:35 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/3024372821' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:20:35.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:35 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/3024372821' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:20:35.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:35 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/3024372821' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:20:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:36 smithi164 ceph-mon[24851]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:36 smithi164 ceph-mon[29512]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:36 smithi172 ceph-mon[25688]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:38 smithi164 ceph-mon[24851]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:38.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:38 smithi164 ceph-mon[29512]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:38.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:38 smithi172 ceph-mon[25688]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:40 smithi164 ceph-mon[24851]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:40 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:20:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:40 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:40 smithi164 ceph-mon[29512]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:40.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:40 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:20:40.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:40 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:40 smithi172 ceph-mon[25688]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:40 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:20:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:40 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:41.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:41 smithi164 ceph-mon[24851]: Deploying daemon osd.4 on smithi172 2024-03-26T02:20:41.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:41 smithi164 ceph-mon[29512]: Deploying daemon osd.4 on smithi172 2024-03-26T02:20:41.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:41 smithi172 ceph-mon[25688]: Deploying daemon osd.4 on smithi172 2024-03-26T02:20:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:42 smithi164 ceph-mon[24851]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:42.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:42 smithi164 ceph-mon[29512]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:42.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:42 smithi172 ceph-mon[25688]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:44.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:44 smithi164 ceph-mon[24851]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:44.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:44 smithi164 ceph-mon[29512]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:44.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:44 smithi172 ceph-mon[25688]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:46.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[24851]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:46.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:46 smithi164 ceph-mon[29512]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:46.548 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:46 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:46.548 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:46 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:46.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:46 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:46.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:46 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:46.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:46 smithi172 ceph-mon[25688]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:48.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:48.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:47 smithi164 ceph-mon[24851]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:48.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:47 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:48.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:48.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:47 smithi164 ceph-mon[29512]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:48.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:47 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:48.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:48.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:47 smithi172 ceph-mon[25688]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:48.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:47 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:48.651 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 4 on host 'smithi172' 2024-03-26T02:20:49.263 DEBUG:teuthology.orchestra.run.smithi172:osd.4> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T02:20:49.265 INFO:tasks.cephadm:Deploying osd.5 on smithi172 with /dev/vg_nvme/lv_3... 2024-03-26T02:20:49.265 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T02:20:49.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[24851]: from='osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:20:49.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[24851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[29512]: from='osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[29512]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:49.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:49 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:49.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:49 smithi172 ceph-mon[25688]: from='osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:20:49.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:49 smithi172 ceph-mon[25688]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:20:49.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:49 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:49.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:49 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:49.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:49 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:49.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:49 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:50.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:50 smithi172 ceph-mon[25688]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:20:50.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:50 smithi172 ceph-mon[25688]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T02:20:50.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:50 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:50.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:50 smithi172 ceph-mon[25688]: from='osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:20:50.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:50 smithi172 ceph-mon[25688]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:20:50.398 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:50 smithi172 ceph-mon[25688]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:50.399 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:20:50 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[29063]: 2024-03-26T02:20:50.241+0000 7f057ed67700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T02:20:50.399 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:20:50 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[29063]: 2024-03-26T02:20:50.245+0000 7f05796fe700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:20:50.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[24851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:20:50.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[24851]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T02:20:50.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:50.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[24851]: from='osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:20:50.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[24851]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:20:50.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[24851]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:50.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[29512]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:20:50.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[29512]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T02:20:50.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:50.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[29512]: from='osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:20:50.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[29512]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:20:50.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:50 smithi164 ceph-mon[29512]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-03-26T02:20:51.522 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:51 smithi172 ceph-mon[25688]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:20:51.522 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:51 smithi172 ceph-mon[25688]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T02:20:51.522 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:51 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:51.522 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:51 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:51.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:51 smithi164 ceph-mon[29512]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:20:51.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:51 smithi164 ceph-mon[29512]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T02:20:51.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:51 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:51.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:51 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:51.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:51 smithi164 ceph-mon[24851]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:20:51.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:51 smithi164 ceph-mon[24851]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T02:20:51.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:51 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:51.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:51 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:52.469 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T02:20:52.469 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T02:20:52.469 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-26T02:20:52.469 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-26T02:20:52.470 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0131753 s, 796 MB/s 2024-03-26T02:20:52.470 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-26T02:20:52.470 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:52 smithi172 ceph-mon[25688]: purged_snaps scrub starts 2024-03-26T02:20:52.470 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:52 smithi172 ceph-mon[25688]: purged_snaps scrub ok 2024-03-26T02:20:52.470 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:52 smithi172 ceph-mon[25688]: osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165] boot 2024-03-26T02:20:52.470 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:52 smithi172 ceph-mon[25688]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T02:20:52.470 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:52 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:52.470 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:52 smithi172 ceph-mon[25688]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:20:52.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[24851]: purged_snaps scrub starts 2024-03-26T02:20:52.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[24851]: purged_snaps scrub ok 2024-03-26T02:20:52.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[24851]: osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165] boot 2024-03-26T02:20:52.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[24851]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T02:20:52.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:52.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[24851]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:20:52.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[29512]: purged_snaps scrub starts 2024-03-26T02:20:52.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[29512]: purged_snaps scrub ok 2024-03-26T02:20:52.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[29512]: osd.4 [v2:172.21.15.172:6800/3407051165,v1:172.21.15.172:6801/3407051165] boot 2024-03-26T02:20:52.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[29512]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T02:20:52.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:20:52.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:52 smithi164 ceph-mon[29512]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:20:53.380 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:53 smithi172 ceph-mon[25688]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T02:20:53.380 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:53 smithi172 ceph-mon[25688]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T02:20:53.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:53 smithi164 ceph-mon[24851]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T02:20:53.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:53 smithi164 ceph-mon[24851]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T02:20:53.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:53 smithi164 ceph-mon[29512]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T02:20:53.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:53 smithi164 ceph-mon[29512]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T02:20:53.791 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_3 2024-03-26T02:20:54.655 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:54 smithi172 ceph-mon[25688]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:20:54.655 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:54 smithi172 ceph-mon[25688]: Detected new or changed devices on smithi172 2024-03-26T02:20:54.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:54.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:54 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi172 to 17088M 2024-03-26T02:20:54.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:54 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.656 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:54 smithi172 ceph-mon[25688]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T02:20:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[24851]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi172 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi172 to 17088M 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[24851]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[29512]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi172 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi172 to 17088M 2024-03-26T02:20:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:20:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:54 smithi164 ceph-mon[29512]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T02:20:56.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[24851]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T02:20:56.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[24851]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T02:20:56.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:56.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:56.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:56.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[29512]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T02:20:56.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[29512]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T02:20:56.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:56.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:56.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:56 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:56.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:56 smithi172 ceph-mon[25688]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T02:20:56.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:56 smithi172 ceph-mon[25688]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T02:20:56.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:56 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:20:56.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:56 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:20:56.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:56 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:57.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:57 smithi164 ceph-mon[29512]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:57.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:57 smithi164 ceph-mon[24851]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:57.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:57 smithi172 ceph-mon[25688]: from='client.34232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:20:58.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[24851]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T02:20:58.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/3830379946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]: dispatch 2024-03-26T02:20:58.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]: dispatch 2024-03-26T02:20:58.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]': finished 2024-03-26T02:20:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[24851]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T02:20:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[29512]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/3830379946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]: dispatch 2024-03-26T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]: dispatch 2024-03-26T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]': finished 2024-03-26T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[29512]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T02:20:58.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:20:58.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:58 smithi172 ceph-mon[25688]: pgmap v113: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T02:20:58.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:58 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/3830379946' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]: dispatch 2024-03-26T02:20:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:58 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]: dispatch 2024-03-26T02:20:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:58 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "deea163e-0adc-4ae4-a720-e0758a614f52"}]': finished 2024-03-26T02:20:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:58 smithi172 ceph-mon[25688]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T02:20:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:20:59.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:20:59 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/2100950642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:20:59.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:20:59 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/2100950642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:20:59.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:20:59 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/2100950642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:21:00.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:00 smithi164 ceph-mon[29512]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T02:21:00.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:21:00.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:00 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:21:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:00 smithi164 ceph-mon[24851]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T02:21:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:21:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:00 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:21:00.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:00 smithi172 ceph-mon[25688]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T02:21:00.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:21:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:00 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:21:02.554 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:02 smithi172 ceph-mon[25688]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T02:21:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:02 smithi164 ceph-mon[24851]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T02:21:02.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:02 smithi164 ceph-mon[29512]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T02:21:04.525 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:04 smithi172 ceph-mon[25688]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:04.525 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:04 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:21:04.526 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:04 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:04.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:04 smithi164 ceph-mon[24851]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:04.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:04 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:21:04.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:04 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:04.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:04 smithi164 ceph-mon[29512]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:04.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:04 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:21:04.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:04 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:05.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:05 smithi172 ceph-mon[25688]: Deploying daemon osd.5 on smithi172 2024-03-26T02:21:05.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:05 smithi164 ceph-mon[24851]: Deploying daemon osd.5 on smithi172 2024-03-26T02:21:05.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:05 smithi164 ceph-mon[29512]: Deploying daemon osd.5 on smithi172 2024-03-26T02:21:06.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:06 smithi164 ceph-mon[24851]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:06.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:06 smithi164 ceph-mon[29512]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:06.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:06 smithi172 ceph-mon[25688]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:08 smithi164 ceph-mon[24851]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:08.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:08 smithi164 ceph-mon[29512]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:08.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:08 smithi172 ceph-mon[25688]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:10.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:10.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:10.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:10.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:10.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[24851]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:10.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:10.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:10.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:10.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:10.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:10 smithi164 ceph-mon[29512]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:10.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:10 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:10.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:10 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:10.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:10 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:10.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:10 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:10.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:10 smithi172 ceph-mon[25688]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:12.237 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 5 on host 'smithi172' 2024-03-26T02:21:12.535 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:12 smithi172 ceph-mon[25688]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:12.535 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:12.535 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:12 smithi172 ceph-mon[25688]: from='osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:21:12.535 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:12 smithi172 ceph-mon[25688]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:21:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:12 smithi164 ceph-mon[24851]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:12.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:12.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:12 smithi164 ceph-mon[24851]: from='osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:21:12.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:12 smithi164 ceph-mon[24851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:21:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:12 smithi164 ceph-mon[29512]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:12 smithi164 ceph-mon[29512]: from='osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:21:12.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:12 smithi164 ceph-mon[29512]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:21:12.848 DEBUG:teuthology.orchestra.run.smithi172:osd.5> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T02:21:12.851 INFO:tasks.cephadm:Deploying osd.6 on smithi172 with /dev/vg_nvme/lv_2... 2024-03-26T02:21:12.851 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T02:21:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:13 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:13 smithi172 ceph-mon[25688]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:21:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:13 smithi172 ceph-mon[25688]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T02:21:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:13 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:14.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:13 smithi172 ceph-mon[25688]: from='osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:14.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:13 smithi172 ceph-mon[25688]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:14.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:13 smithi172 ceph-mon[25688]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:14.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[24851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[24851]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[24851]: from='osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[24851]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[24851]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[29512]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[29512]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[29512]: from='osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[29512]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:13 smithi164 ceph-mon[29512]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:21:14.557 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:21:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[32014]: 2024-03-26T02:21:14.252+0000 7f5bfc347700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T02:21:14.557 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:21:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[32014]: 2024-03-26T02:21:14.259+0000 7f5bf4cda700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:21:15.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:15 smithi164 ceph-mon[24851]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:21:15.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:15 smithi164 ceph-mon[24851]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:21:15.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:15 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:15.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:15 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:15.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:15 smithi164 ceph-mon[29512]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:21:15.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:15 smithi164 ceph-mon[29512]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:21:15.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:15 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:15.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:15 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:15.554 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:15 smithi172 ceph-mon[25688]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:21:15.554 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:15 smithi172 ceph-mon[25688]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:21:15.554 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:15 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:15.554 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:15 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:16.427 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:16 smithi172 ceph-mon[25688]: purged_snaps scrub starts 2024-03-26T02:21:16.427 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:16 smithi172 ceph-mon[25688]: purged_snaps scrub ok 2024-03-26T02:21:16.427 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:16 smithi172 ceph-mon[25688]: osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481] boot 2024-03-26T02:21:16.427 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:16 smithi172 ceph-mon[25688]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T02:21:16.427 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:16 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:16.427 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:16 smithi172 ceph-mon[25688]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:16.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[24851]: purged_snaps scrub starts 2024-03-26T02:21:16.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[24851]: purged_snaps scrub ok 2024-03-26T02:21:16.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[24851]: osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481] boot 2024-03-26T02:21:16.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[24851]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T02:21:16.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:16.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[24851]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:16.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[29512]: purged_snaps scrub starts 2024-03-26T02:21:16.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[29512]: purged_snaps scrub ok 2024-03-26T02:21:16.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[29512]: osd.5 [v2:172.21.15.172:6808/887601481,v1:172.21.15.172:6809/887601481] boot 2024-03-26T02:21:16.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[29512]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T02:21:16.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:21:16.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:16 smithi164 ceph-mon[29512]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:17.237 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T02:21:17.237 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T02:21:17.237 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-26T02:21:17.237 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-26T02:21:17.237 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133438 s, 786 MB/s 2024-03-26T02:21:17.237 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-26T02:21:17.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:17.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:17.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:17.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:17.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:17.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:17 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:17 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:17.873 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_2 2024-03-26T02:21:18.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:18 smithi172 ceph-mon[25688]: Detected new or changed devices on smithi172 2024-03-26T02:21:18.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:18 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi172 to 8544M 2024-03-26T02:21:18.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:18 smithi172 ceph-mon[25688]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:21:18.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:18 smithi172 ceph-mon[25688]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:18.485 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:18 smithi172 ceph-mon[25688]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:21:18.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi172 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi172 to 8544M 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[24851]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[24851]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[24851]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi172 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi172 to 8544M 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[29512]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[29512]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:18 smithi164 ceph-mon[29512]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:21:19.399 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:19 smithi172 ceph-mon[25688]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T02:21:19.399 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:19 smithi172 ceph-mon[25688]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:21:19.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:19 smithi164 ceph-mon[24851]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T02:21:19.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:19 smithi164 ceph-mon[24851]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:21:19.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:19 smithi164 ceph-mon[29512]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T02:21:19.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:19 smithi164 ceph-mon[29512]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:21:20.536 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:20 smithi172 ceph-mon[25688]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:20.536 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:20 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:21:20.536 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:20 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:21:20.536 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:20 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:20 smithi164 ceph-mon[24851]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:20 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:21:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:20 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:21:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:20 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:20 smithi164 ceph-mon[29512]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:20 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:21:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:20 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:21:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:20 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:21.474 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:21 smithi172 ceph-mon[25688]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:21:21.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:21 smithi164 ceph-mon[24851]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:21:21.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:21 smithi164 ceph-mon[29512]: from='client.34259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:21:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[24851]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T02:21:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/3434459252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]: dispatch 2024-03-26T02:21:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]: dispatch 2024-03-26T02:21:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]': finished 2024-03-26T02:21:22.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[24851]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T02:21:22.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:22.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[29512]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T02:21:22.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/3434459252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]: dispatch 2024-03-26T02:21:22.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]: dispatch 2024-03-26T02:21:22.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]': finished 2024-03-26T02:21:22.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[29512]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T02:21:22.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:22 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:22 smithi172 ceph-mon[25688]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-03-26T02:21:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:22 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/3434459252' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]: dispatch 2024-03-26T02:21:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:22 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]: dispatch 2024-03-26T02:21:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:22 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "83914243-cedc-4cfe-9b0b-4dfc37d72d23"}]': finished 2024-03-26T02:21:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:22 smithi172 ceph-mon[25688]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T02:21:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:22 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:23.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:23 smithi164 ceph-mon[24851]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T02:21:23.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:23 smithi164 ceph-mon[24851]: Cluster is now healthy 2024-03-26T02:21:23.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:23 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/2992120070' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:21:23.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:23 smithi164 ceph-mon[29512]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T02:21:23.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:23 smithi164 ceph-mon[29512]: Cluster is now healthy 2024-03-26T02:21:23.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:23 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/2992120070' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:21:23.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:23 smithi172 ceph-mon[25688]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T02:21:23.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:23 smithi172 ceph-mon[25688]: Cluster is now healthy 2024-03-26T02:21:23.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:23 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/2992120070' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:21:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:24 smithi172 ceph-mon[25688]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T02:21:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:24 smithi164 ceph-mon[24851]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T02:21:24.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:24 smithi164 ceph-mon[29512]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-03-26T02:21:26.695 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:26 smithi172 ceph-mon[25688]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T02:21:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:26 smithi164 ceph-mon[24851]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T02:21:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:26 smithi164 ceph-mon[29512]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T02:21:28.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:28 smithi172 ceph-mon[25688]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-26T02:21:28.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:28 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:21:28.777 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:28 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:28 smithi164 ceph-mon[24851]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-26T02:21:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:28 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:21:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:28 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:28 smithi164 ceph-mon[29512]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-26T02:21:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:28 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:21:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:28 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:29.781 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:29 smithi172 ceph-mon[25688]: Deploying daemon osd.6 on smithi172 2024-03-26T02:21:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:29 smithi164 ceph-mon[24851]: Deploying daemon osd.6 on smithi172 2024-03-26T02:21:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:29 smithi164 ceph-mon[29512]: Deploying daemon osd.6 on smithi172 2024-03-26T02:21:30.645 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:30 smithi172 ceph-mon[25688]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T02:21:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:30 smithi164 ceph-mon[24851]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T02:21:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:30 smithi164 ceph-mon[29512]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T02:21:32.731 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:32 smithi172 ceph-mon[25688]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:32 smithi164 ceph-mon[24851]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:32.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:32 smithi164 ceph-mon[29512]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:34.667 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:34 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:34.667 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:34 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:34.667 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:34 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:34.667 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:34 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:34.667 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:34 smithi172 ceph-mon[25688]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[24851]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:34 smithi164 ceph-mon[29512]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:36.416 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 6 on host 'smithi172' 2024-03-26T02:21:36.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:36 smithi172 ceph-mon[25688]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:36.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:36 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:36.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:36 smithi172 ceph-mon[25688]: from='osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:21:36.745 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:36 smithi172 ceph-mon[25688]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:21:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:36 smithi164 ceph-mon[29512]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:36 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:36 smithi164 ceph-mon[29512]: from='osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:21:36.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:36 smithi164 ceph-mon[29512]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:21:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:36 smithi164 ceph-mon[24851]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:36 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:36 smithi164 ceph-mon[24851]: from='osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:21:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:36 smithi164 ceph-mon[24851]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:21:37.077 DEBUG:teuthology.orchestra.run.smithi172:osd.6> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T02:21:37.079 INFO:tasks.cephadm:Deploying osd.7 on smithi172 with /dev/vg_nvme/lv_1... 2024-03-26T02:21:37.080 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T02:21:38.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[29512]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[29512]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[29512]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[29512]: from='osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[29512]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[24851]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[24851]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:21:38.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[24851]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T02:21:38.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:38.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[24851]: from='osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:38.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:38 smithi164 ceph-mon[24851]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:38.403 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:38.403 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-mon[25688]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:38.403 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-mon[25688]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:21:38.403 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-mon[25688]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T02:21:38.403 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:38.403 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-mon[25688]: from='osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:38.403 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-mon[25688]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:21:38.740 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[34908]: 2024-03-26T02:21:38.423+0000 7fb3499dd700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T02:21:38.740 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:21:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[34908]: 2024-03-26T02:21:38.431+0000 7fb342b71700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:21:39.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:39 smithi172 ceph-mon[25688]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:21:39.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:39 smithi172 ceph-mon[25688]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:21:39.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:39 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:39.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:39 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:39 smithi164 ceph-mon[24851]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:21:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:39 smithi164 ceph-mon[24851]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:21:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:39 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:39.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:39 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:39 smithi164 ceph-mon[29512]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:21:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:39 smithi164 ceph-mon[29512]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:21:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:39 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:39 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:40.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:40 smithi172 ceph-mon[25688]: purged_snaps scrub starts 2024-03-26T02:21:40.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:40 smithi172 ceph-mon[25688]: purged_snaps scrub ok 2024-03-26T02:21:40.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:40 smithi172 ceph-mon[25688]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:40.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:40 smithi172 ceph-mon[25688]: osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954] boot 2024-03-26T02:21:40.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:40 smithi172 ceph-mon[25688]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T02:21:40.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:40 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:40.620 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T02:21:40.621 INFO:teuthology.orchestra.run.smithi172.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T02:21:40.621 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10+0 records in 2024-03-26T02:21:40.621 INFO:teuthology.orchestra.run.smithi172.stderr:10+0 records out 2024-03-26T02:21:40.621 INFO:teuthology.orchestra.run.smithi172.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.015047 s, 697 MB/s 2024-03-26T02:21:40.621 INFO:teuthology.orchestra.run.smithi172.stderr:--> Zapping successful for: 2024-03-26T02:21:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[24851]: purged_snaps scrub starts 2024-03-26T02:21:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[24851]: purged_snaps scrub ok 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[24851]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[24851]: osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954] boot 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[24851]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[29512]: purged_snaps scrub starts 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[29512]: purged_snaps scrub ok 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[29512]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[29512]: osd.6 [v2:172.21.15.172:6816/928726954,v1:172.21.15.172:6817/928726954] boot 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[29512]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T02:21:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:40 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:21:41.446 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi172:vg_nvme/lv_1 2024-03-26T02:21:41.756 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:41 smithi172 ceph-mon[25688]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T02:21:41.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:41 smithi164 ceph-mon[24851]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T02:21:41.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:41 smithi164 ceph-mon[29512]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T02:21:42.770 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:42.770 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:42.771 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:42 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:21:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:21:42.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:42.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:42.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:42.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:42.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:42 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:43.789 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:43 smithi172 ceph-mon[25688]: Detected new or changed devices on smithi172 2024-03-26T02:21:43.789 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:43 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi172 to 5696M 2024-03-26T02:21:43.790 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:43 smithi172 ceph-mon[25688]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:21:43.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:43 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi172 2024-03-26T02:21:43.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:43 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi172 to 5696M 2024-03-26T02:21:43.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:43 smithi164 ceph-mon[24851]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:21:43.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:43 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi172 2024-03-26T02:21:43.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:43 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi172 to 5696M 2024-03-26T02:21:43.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:43 smithi164 ceph-mon[29512]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:21:44.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:44 smithi172 ceph-mon[25688]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:44.788 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:44 smithi172 ceph-mon[25688]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:21:44.789 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:44 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:21:44.789 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:44 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:21:44.789 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:44 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[24851]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[24851]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:21:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:21:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:21:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[29512]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[29512]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:21:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:21:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:21:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:44 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:45 smithi164 ceph-mon[24851]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:21:45.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:45 smithi164 ceph-mon[29512]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:21:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:45 smithi172 ceph-mon[25688]: from='client.34286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi172:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:21:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[24851]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/4132648445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]: dispatch 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]: dispatch 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[24851]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]': finished 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[24851]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[29512]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/4132648445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]: dispatch 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]: dispatch 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[29512]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]': finished 2024-03-26T02:21:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[29512]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T02:21:46.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:46 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:21:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:46 smithi172 ceph-mon[25688]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-26T02:21:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:46 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/4132648445' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]: dispatch 2024-03-26T02:21:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:46 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]: dispatch 2024-03-26T02:21:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:46 smithi172 ceph-mon[25688]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcd7fb7d-3b97-4337-8264-8312679cd9d9"}]': finished 2024-03-26T02:21:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:46 smithi172 ceph-mon[25688]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T02:21:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:46 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:21:47.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:47 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/412329580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:21:47.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:47 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/412329580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:21:47.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:47 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/412329580' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:21:48.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:48 smithi164 ceph-mon[24851]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-26T02:21:48.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:48 smithi164 ceph-mon[29512]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-26T02:21:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:48 smithi172 ceph-mon[25688]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-03-26T02:21:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:50 smithi164 ceph-mon[24851]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T02:21:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:50 smithi164 ceph-mon[29512]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T02:21:50.804 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:50 smithi172 ceph-mon[25688]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T02:21:52.729 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:52 smithi172 ceph-mon[25688]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T02:21:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:52 smithi164 ceph-mon[24851]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T02:21:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:52 smithi164 ceph-mon[29512]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T02:21:53.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:53 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:21:53.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:53 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:53.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:53 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:21:53.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:53 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:53.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:53 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:21:53.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:53 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:54.742 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:54 smithi172 ceph-mon[25688]: Deploying daemon osd.7 on smithi172 2024-03-26T02:21:54.743 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:54 smithi172 ceph-mon[25688]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T02:21:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:54 smithi164 ceph-mon[24851]: Deploying daemon osd.7 on smithi172 2024-03-26T02:21:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:54 smithi164 ceph-mon[24851]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T02:21:54.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:54 smithi164 ceph-mon[29512]: Deploying daemon osd.7 on smithi172 2024-03-26T02:21:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:54 smithi164 ceph-mon[29512]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-03-26T02:21:56.764 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:56 smithi172 ceph-mon[25688]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:56 smithi164 ceph-mon[24851]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:56.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:56 smithi164 ceph-mon[29512]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:58.588 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:58 smithi172 ceph-mon[25688]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:58.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:58.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:58.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:58.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:58 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[24851]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[29512]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:58.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:58 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:59.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:59 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:21:59.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:21:59 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:21:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:59 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:21:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:21:59 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:21:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:59 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:21:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:21:59 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:22:00.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:00 smithi172 ceph-mon[25688]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:00 smithi164 ceph-mon[24851]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:00.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:00 smithi164 ceph-mon[29512]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:01.394 INFO:teuthology.orchestra.run.smithi172.stdout:Created osd(s) 7 on host 'smithi172' 2024-03-26T02:22:01.734 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:01 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:01.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:01 smithi172 ceph-mon[25688]: from='osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:22:01.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:01 smithi172 ceph-mon[25688]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:22:01.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:01 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:01.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:01 smithi172 ceph-mon[25688]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:02.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:02.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[24851]: from='osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:22:02.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[24851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:22:02.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:02.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[24851]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:02.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:02.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[29512]: from='osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:22:02.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[29512]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:22:02.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:02.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:01 smithi164 ceph-mon[29512]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:02.069 DEBUG:teuthology.orchestra.run.smithi172:osd.7> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T02:22:02.071 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T02:22:02.071 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T02:22:02.568 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:02 smithi172 ceph-mon[25688]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:22:02.568 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:02 smithi172 ceph-mon[25688]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T02:22:02.568 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:02 smithi172 ceph-mon[25688]: from='osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:22:02.568 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:02 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:02.568 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:02 smithi172 ceph-mon[25688]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:22:02.568 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:22:02 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[37834]: 2024-03-26T02:22:02.556+0000 7f65cf785700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T02:22:02.568 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:22:02 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[37834]: 2024-03-26T02:22:02.566+0000 7f65ca11c700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:22:02.692 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[24851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:22:02.692 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[24851]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T02:22:02.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[24851]: from='osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:22:02.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:02.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[24851]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:22:02.693 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[29512]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:22:02.693 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[29512]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T02:22:02.693 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[29512]: from='osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:22:02.693 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:02.693 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:02 smithi164 ceph-mon[29512]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:22:03.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[24851]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:22:03.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[24851]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T02:22:03.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[24851]: osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110] boot 2024-03-26T02:22:03.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[24851]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T02:22:03.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[24851]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:03.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[29512]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:22:03.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[29512]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T02:22:03.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[29512]: osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110] boot 2024-03-26T02:22:03.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[29512]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T02:22:03.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:03 smithi164 ceph-mon[29512]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:03 smithi172 ceph-mon[25688]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]': finished 2024-03-26T02:22:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:03 smithi172 ceph-mon[25688]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T02:22:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:03 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:03 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:03 smithi172 ceph-mon[25688]: osd.7 [v2:172.21.15.172:6824/3771214110,v1:172.21.15.172:6825/3771214110] boot 2024-03-26T02:22:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:03 smithi172 ceph-mon[25688]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T02:22:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:03 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:03 smithi172 ceph-mon[25688]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:22:04.051 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:22:04.745 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:04 smithi164 ceph-mon[24851]: purged_snaps scrub starts 2024-03-26T02:22:04.745 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:04 smithi164 ceph-mon[24851]: purged_snaps scrub ok 2024-03-26T02:22:04.745 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:04 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/638270825' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:22:04.745 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:04 smithi164 ceph-mon[24851]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T02:22:04.745 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:04 smithi164 ceph-mon[29512]: purged_snaps scrub starts 2024-03-26T02:22:04.745 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:04 smithi164 ceph-mon[29512]: purged_snaps scrub ok 2024-03-26T02:22:04.745 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:04 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/638270825' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:22:04.745 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:04 smithi164 ceph-mon[29512]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T02:22:04.767 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1711419723,"num_in_osds":8,"osd_in_since":1711419706,"num_remapped_pgs":1} 2024-03-26T02:22:04.768 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:22:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:04 smithi172 ceph-mon[25688]: purged_snaps scrub starts 2024-03-26T02:22:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:04 smithi172 ceph-mon[25688]: purged_snaps scrub ok 2024-03-26T02:22:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:04 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/638270825' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:22:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:04 smithi172 ceph-mon[25688]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T02:22:04.992 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:06.184 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi172 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi172 to 4272M 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:06.185 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[24851]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi172 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.186 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi172 to 4272M 2024-03-26T02:22:06.187 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.187 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.187 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:06.187 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:06.187 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:06.187 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.187 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T02:22:06.187 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:05 smithi164 ceph-mon[29512]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: Detected new or changed devices on smithi172 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi172 to 4272M 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:06.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:06.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:06.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:06.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T02:22:06.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:05 smithi172 ceph-mon[25688]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:06.671 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:22:06.671 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":57,"fsid":"e2654d3c-eb16-11ee-95d0-87774f69a715","created":"2024-03-26T02:17:14.051039+0000","modified":"2024-03-26T02:22:06.245781+0000","last_up_change":"2024-03-26T02:22:03.237622+0000","last_in_change":"2024-03-26T02:21:46.121780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:20:07.678670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fa926af6-17d0-4176-b716-b668e463738d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6803","nonce":378049558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6805","nonce":378049558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6809","nonce":378049558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6807","nonce":378049558}]},"public_addr":"172.21.15.164:6803/378049558","cluster_addr":"172.21.15.164:6805/378049558","heartbeat_back_addr":"172.21.15.164:6809/378049558","heartbeat_front_addr":"172.21.15.164:6807/378049558","state":["exists","up"]},{"osd":1,"uuid":"662f7218-b733-4bd5-9c05-305d11ec6b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6811","nonce":3337400680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6813","nonce":3337400680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6817","nonce":3337400680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6815","nonce":3337400680}]},"public_addr":"172.21.15.164:6811/3337400680","cluster_addr":"172.21.15.164:6813/3337400680","heartbeat_back_addr":"172.21.15.164:6817/3337400680","heartbeat_front_addr":"172.21.15.164:6815/3337400680","state":["exists","up"]},{"osd":2,"uuid":"cea9695c-f306-468a-8071-debc4326b99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6819","nonce":851694553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6821","nonce":851694553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6825","nonce":851694553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6823","nonce":851694553}]},"public_addr":"172.21.15.164:6819/851694553","cluster_addr":"172.21.15.164:6821/851694553","heartbeat_back_addr":"172.21.15.164:6825/851694553","heartbeat_front_addr":"172.21.15.164:6823/851694553","state":["exists","up"]},{"osd":3,"uuid":"b90e1e0c-0eee-46e3-b56c-54e8aea3bb75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6827","nonce":3222350232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6829","nonce":3222350232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6833","nonce":3222350232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6831","nonce":3222350232}]},"public_addr":"172.21.15.164:6827/3222350232","cluster_addr":"172.21.15.164:6829/3222350232","heartbeat_back_addr":"172.21.15.164:6833/3222350232","heartbeat_front_addr":"172.21.15.164:6831/3222350232","state":["exists","up"]},{"osd":4,"uuid":"f73412ca-7838-48e2-a5dc-897dccf6b4bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6801","nonce":3407051165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6803","nonce":3407051165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6807","nonce":3407051165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6805","nonce":3407051165}]},"public_addr":"172.21.15.172:6801/3407051165","cluster_addr":"172.21.15.172:6803/3407051165","heartbeat_back_addr":"172.21.15.172:6807/3407051165","heartbeat_front_addr":"172.21.15.172:6805/3407051165","state":["exists","up"]},{"osd":5,"uuid":"deea163e-0adc-4ae4-a720-e0758a614f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6809","nonce":887601481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6811","nonce":887601481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6815","nonce":887601481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6813","nonce":887601481}]},"public_addr":"172.21.15.172:6809/887601481","cluster_addr":"172.21.15.172:6811/887601481","heartbeat_back_addr":"172.21.15.172:6815/887601481","heartbeat_front_addr":"172.21.15.172:6813/887601481","state":["exists","up"]},{"osd":6,"uuid":"83914243-cedc-4cfe-9b0b-4dfc37d72d23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6817","nonce":928726954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6819","nonce":928726954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6823","nonce":928726954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6821","nonce":928726954}]},"public_addr":"172.21.15.172:6817/928726954","cluster_addr":"172.21.15.172:6819/928726954","heartbeat_back_addr":"172.21.15.172:6823/928726954","heartbeat_front_addr":"172.21.15.172:6821/928726954","state":["exists","up"]},{"osd":7,"uuid":"fcd7fb7d-3b97-4337-8264-8312679cd9d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6825","nonce":3771214110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6827","nonce":3771214110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6831","nonce":3771214110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6829","nonce":3771214110}]},"public_addr":"172.21.15.172:6825/3771214110","cluster_addr":"172.21.15.172:6827/3771214110","heartbeat_back_addr":"172.21.15.172:6831/3771214110","heartbeat_front_addr":"172.21.15.172:6829/3771214110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:19:17.847711+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:19:41.445819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:04.505257+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:27.790127+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:49.467227+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:21:13.368554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:21:37.449411+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:22:02.106264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:6800/1272665921":"2024-03-27T02:17:59.295370+0000","172.21.15.164:0/1798451206":"2024-03-27T02:17:59.295370+0000","172.21.15.164:0/3009346081":"2024-03-27T02:17:59.295370+0000","172.21.15.164:0/3498887725":"2024-03-27T02:17:37.995264+0000","172.21.15.164:0/1487529581":"2024-03-27T02:17:37.995264+0000","172.21.15.164:0/3486476943":"2024-03-27T02:17:59.295370+0000","172.21.15.164:6801/1272665921":"2024-03-27T02:17:59.295370+0000","172.21.15.164:6801/2862881695":"2024-03-27T02:17:37.995264+0000","172.21.15.164:6800/2862881695":"2024-03-27T02:17:37.995264+0000","172.21.15.164:0/75066052":"2024-03-27T02:17:37.995264+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:22:07.385 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T02:20:07.678670+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-26T02:22:07.386 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T02:22:07.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:07 smithi164 ceph-mon[24851]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T02:22:07.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:07 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2011553983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:07.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:07 smithi164 ceph-mon[29512]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T02:22:07.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:07 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2011553983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:07.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:07 smithi172 ceph-mon[25688]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T02:22:07.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:07 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2011553983' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:07.600 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:08.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:08 smithi164 ceph-mon[24851]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:22:08.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:08 smithi164 ceph-mon[24851]: pgmap v172: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T02:22:08.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:08 smithi164 ceph-mon[29512]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:22:08.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:08 smithi164 ceph-mon[29512]: pgmap v172: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T02:22:08.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:08 smithi172 ceph-mon[25688]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:22:08.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:08 smithi172 ceph-mon[25688]: pgmap v172: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-03-26T02:22:09.484 INFO:teuthology.orchestra.run.smithi164.stdout:pg_num: 1 2024-03-26T02:22:10.175 INFO:tasks.cephadm:Adding prometheus.a on smithi172 2024-03-26T02:22:10.175 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi172=a' 2024-03-26T02:22:10.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:10 smithi172 ceph-mon[25688]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T02:22:10.493 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:10 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1283534233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:22:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:10 smithi164 ceph-mon[24851]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T02:22:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:10 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1283534233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:22:10.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:10 smithi164 ceph-mon[29512]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-03-26T02:22:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:10 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1283534233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:22:12.143 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled prometheus update... 2024-03-26T02:22:12.758 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:12 smithi172 ceph-mon[25688]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-26T02:22:12.758 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:12.758 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:12.758 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:12.758 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:12.758 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:12.758 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:12 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:22:12.783 DEBUG:teuthology.orchestra.run.smithi172:prometheus.a> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T02:22:12.785 INFO:tasks.cephadm:Adding node-exporter.a on smithi164 2024-03-26T02:22:12.785 INFO:tasks.cephadm:Adding node-exporter.b on smithi172 2024-03-26T02:22:12.785 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi164=a;smithi172=b' 2024-03-26T02:22:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[24851]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-26T02:22:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[29512]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' 2024-03-26T02:22:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:12 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:22:13.460 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ignoring --setuser ceph since I am not root 2024-03-26T02:22:13.527 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ignoring --setgroup ceph since I am not root 2024-03-26T02:22:13.527 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-mgr[26881]: -- 172.21.15.172:0/857254550 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555e4efd8340 con 0x555e4efb0800 2024-03-26T02:22:13.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mon[29512]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:13.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mon[29512]: Saving service prometheus spec with placement smithi172=a;count:1 2024-03-26T02:22:13.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mon[29512]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:22:13.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mon[29512]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:22:13.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mon[24851]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:13.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mon[24851]: Saving service prometheus spec with placement smithi172=a;count:1 2024-03-26T02:22:13.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mon[24851]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:22:13.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mon[24851]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:22:13.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ignoring --setuser ceph since I am not root 2024-03-26T02:22:13.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ignoring --setgroup ceph since I am not root 2024-03-26T02:22:13.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-mgr[25091]: -- 172.21.15.164:0/1006613479 <== mon.1 v2:172.21.15.164:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5639b7c36340 con 0x5639b7c0ec00 2024-03-26T02:22:13.786 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:13.693+0000 7f6065a2f000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:22:13.786 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-mon[25688]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:13.786 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-mon[25688]: Saving service prometheus spec with placement smithi172=a;count:1 2024-03-26T02:22:13.786 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-mon[25688]: from='mgr.14148 172.21.15.164:0/2460726932' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:22:13.787 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-mon[25688]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:22:14.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:13.698+0000 7f6c219ad000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:22:14.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:13.788+0000 7f6c219ad000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:22:14.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:13.785+0000 7f6065a2f000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:22:14.418 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:14.386+0000 7f6065a2f000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:22:14.701 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:14.380+0000 7f6c219ad000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:22:14.701 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:14.468+0000 7f6c219ad000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:22:14.701 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:14.552+0000 7f6c219ad000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:22:14.724 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:14.482+0000 7f6065a2f000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:22:14.724 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:14.567+0000 7f6065a2f000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:22:14.724 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:14.723+0000 7f6065a2f000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:22:14.972 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:14.701+0000 7f6c219ad000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:22:14.972 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:14.781+0000 7f6c219ad000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:22:14.972 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:14.889+0000 7f6c219ad000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:22:14.972 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:14.971+0000 7f6c219ad000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:22:15.002 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:14.806+0000 7f6065a2f000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:22:15.002 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:14.918+0000 7f6065a2f000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:22:15.002 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:15.001+0000 7f6065a2f000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:22:15.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:15.134+0000 7f6c219ad000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:22:15.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:15.219+0000 7f6c219ad000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:22:15.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:15.164+0000 7f6065a2f000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:22:15.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:15.248+0000 7f6065a2f000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:22:16.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:16.240+0000 7f6c219ad000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:22:16.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:16.368+0000 7f6c219ad000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:22:16.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:16.280+0000 7f6065a2f000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:22:16.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:16.410+0000 7f6065a2f000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:22:17.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:16.559+0000 7f6c219ad000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:22:17.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:16.646+0000 7f6c219ad000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:22:17.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:16.782+0000 7f6c219ad000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:22:17.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:16.601+0000 7f6065a2f000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:22:17.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:16.688+0000 7f6065a2f000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:22:17.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:16.823+0000 7f6065a2f000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:22:17.422 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:17.105+0000 7f6c219ad000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:22:17.422 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:17.324+0000 7f6c219ad000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:22:17.450 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:17.142+0000 7f6065a2f000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:22:17.450 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:17.363+0000 7f6065a2f000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:22:17.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:17.421+0000 7f6c219ad000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:22:17.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:17.604+0000 7f6c219ad000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:22:17.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:17.449+0000 7f6065a2f000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:22:17.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:17.619+0000 7f6065a2f000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:22:18.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:18.530+0000 7f6c219ad000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:22:18.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:18.536+0000 7f6065a2f000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:22:19.096 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:18.834+0000 7f6065a2f000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:22:19.097 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:18.929+0000 7f6065a2f000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:22:19.097 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:19.079+0000 7f6065a2f000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:22:19.097 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: [26/Mar/2024:02:22:19] ENGINE Bus STARTING 2024-03-26T02:22:19.244 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:18.831+0000 7f6c219ad000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:22:19.244 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:18.928+0000 7f6c219ad000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:22:19.244 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:19.081+0000 7f6c219ad000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:22:19.352 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: Active manager daemon y restarted 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: Activating manager daemon y 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: Standby manager daemon x restarted 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: Standby manager daemon x started 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: CherryPy Checker: 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: The Application mounted at '' has an empty config. 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: [26/Mar/2024:02:22:19] ENGINE Serving on http://:::9283 2024-03-26T02:22:19.353 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: [26/Mar/2024:02:22:19] ENGINE Bus STARTED 2024-03-26T02:22:19.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:22:19] ENGINE Bus STARTING 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: Active manager daemon y restarted 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: Activating manager daemon y 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: Standby manager daemon x restarted 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: Standby manager daemon x started 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: Active manager daemon y restarted 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: Activating manager daemon y 2024-03-26T02:22:19.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: Standby manager daemon x restarted 2024-03-26T02:22:19.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: Standby manager daemon x started 2024-03-26T02:22:19.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:22:19.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:22:19.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:22:19.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:22:19.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3506924625' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:22:19.901 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: CherryPy Checker: 2024-03-26T02:22:19.901 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: The Application mounted at '' has an empty config. 2024-03-26T02:22:19.901 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:22:19.901 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:22:19] ENGINE Serving on http://:::9283 2024-03-26T02:22:19.901 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:22:19] ENGINE Bus STARTED 2024-03-26T02:22:19.901 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:22:19] ENGINE Bus STARTING 2024-03-26T02:22:19.901 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:22:19] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:22:19.901 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:22:19] ENGINE Bus STARTED 2024-03-26T02:22:20.269 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled node-exporter update... 2024-03-26T02:22:20.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: mgrmap e17: y(active, starting, since 0.156712s), standbys: x 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: Manager daemon y is now available 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: [26/Mar/2024:02:22:19] ENGINE Bus STARTING 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: [26/Mar/2024:02:22:19] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: [26/Mar/2024:02:22:19] ENGINE Bus STARTED 2024-03-26T02:22:20.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: mgrmap e17: y(active, starting, since 0.156712s), standbys: x 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:22:20.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: Manager daemon y is now available 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: [26/Mar/2024:02:22:19] ENGINE Bus STARTING 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: [26/Mar/2024:02:22:19] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:22:20.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: [26/Mar/2024:02:22:19] ENGINE Bus STARTED 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: mgrmap e17: y(active, starting, since 0.156712s), standbys: x 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:22:20.549 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: Manager daemon y is now available 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:20.550 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:20.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:22:20.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:22:20.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: [26/Mar/2024:02:22:19] ENGINE Bus STARTING 2024-03-26T02:22:20.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: [26/Mar/2024:02:22:19] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:22:20.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: [26/Mar/2024:02:22:19] ENGINE Bus STARTED 2024-03-26T02:22:20.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:20.931 DEBUG:teuthology.orchestra.run.smithi164:node-exporter.a> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T02:22:20.934 DEBUG:teuthology.orchestra.run.smithi172:node-exporter.b> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T02:22:20.936 INFO:tasks.cephadm:Adding alertmanager.a on smithi164 2024-03-26T02:22:20.936 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi164=a' 2024-03-26T02:22:21.505 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[24851]: mgrmap e18: y(active, since 1.16045s), standbys: x 2024-03-26T02:22:21.505 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[24851]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi164=a;smithi172=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:21.505 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[24851]: Saving service node-exporter spec with placement smithi164=a;smithi172=b;count:2 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[24851]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[29512]: mgrmap e18: y(active, since 1.16045s), standbys: x 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[29512]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi164=a;smithi172=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[29512]: Saving service node-exporter spec with placement smithi164=a;smithi172=b;count:2 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[29512]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:21.506 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:21 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:21.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:21 smithi172 ceph-mon[25688]: mgrmap e18: y(active, since 1.16045s), standbys: x 2024-03-26T02:22:21.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:21 smithi172 ceph-mon[25688]: from='client.34349 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi164=a;smithi172=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:21.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:21 smithi172 ceph-mon[25688]: Saving service node-exporter spec with placement smithi164=a;smithi172=b;count:2 2024-03-26T02:22:21.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:21 smithi172 ceph-mon[25688]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:21.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:21 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:21.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:21 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:22.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:22 smithi172 ceph-mon[25688]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:22.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:22 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:22.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:22 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:22:22.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:22 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:22:22.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:22 smithi172 ceph-mon[25688]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-26T02:22:22.735 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:22 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:22.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[29512]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[29512]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[24851]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:22:22.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[24851]: mgrmap e19: y(active, since 2s), standbys: x 2024-03-26T02:22:22.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:22 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:23.138 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled alertmanager update... 2024-03-26T02:22:23.856 DEBUG:teuthology.orchestra.run.smithi164:alertmanager.a> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T02:22:23.858 INFO:tasks.cephadm:Adding grafana.a on smithi172 2024-03-26T02:22:23.859 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi172=a' 2024-03-26T02:22:24.093 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:24.094 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:24 smithi172 ceph-mon[25688]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:24.094 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:24 smithi172 ceph-mon[25688]: Saving service alertmanager spec with placement smithi164=a;count:1 2024-03-26T02:22:24.094 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:24.094 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:24.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:24.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[29512]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[29512]: Saving service alertmanager spec with placement smithi164=a;count:1 2024-03-26T02:22:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[24851]: from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi164=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[24851]: Saving service alertmanager spec with placement smithi164=a;count:1 2024-03-26T02:22:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:24.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:25.160 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:25 smithi172 ceph-mon[25688]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:25.161 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:25 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:22:25.161 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:25 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:22:25.161 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:25 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:25.161 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:25 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:25.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[24851]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[29512]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:25.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:25 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:26.007 INFO:teuthology.orchestra.run.smithi172.stdout:Scheduled grafana update... 2024-03-26T02:22:26.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:26 smithi172 ceph-mon[25688]: Deploying daemon prometheus.a on smithi172 2024-03-26T02:22:26.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:26 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:26.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:26 smithi164 ceph-mon[29512]: Deploying daemon prometheus.a on smithi172 2024-03-26T02:22:26.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:26 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:26.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:26 smithi164 ceph-mon[24851]: Deploying daemon prometheus.a on smithi172 2024-03-26T02:22:26.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:26 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:26.946 DEBUG:teuthology.orchestra.run.smithi172:grafana.a> sudo journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T02:22:26.948 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T02:22:26.951 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:22:27.263 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:27 smithi164 ceph-mon[29512]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:27.263 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:27 smithi164 ceph-mon[29512]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:27.263 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:27 smithi164 ceph-mon[29512]: Saving service grafana spec with placement smithi172=a;count:1 2024-03-26T02:22:27.263 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:27 smithi164 ceph-mon[24851]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:27.263 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:27 smithi164 ceph-mon[24851]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:27.263 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:27 smithi164 ceph-mon[24851]: Saving service grafana spec with placement smithi172=a;count:1 2024-03-26T02:22:27.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:27 smithi172 ceph-mon[25688]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:27.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:27 smithi172 ceph-mon[25688]: from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi172=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:22:27.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:27 smithi172 ceph-mon[25688]: Saving service grafana spec with placement smithi172=a;count:1 2024-03-26T02:22:28.943 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2024-03-26T02:22:28.943 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBkMQJm/gr9NxAAaQ8vdy0MnKdrNZ9rVeB5/g== 2024-03-26T02:22:29.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:29 smithi164 ceph-mon[24851]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:29.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:29 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3654255578' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:29.266 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:29 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3654255578' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:22:29.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:29 smithi164 ceph-mon[29512]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:29.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:29 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3654255578' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:29.266 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:29 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3654255578' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:22:29.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:29 smithi172 ceph-mon[25688]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:29.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:29 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3654255578' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:29.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:29 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3654255578' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:22:29.685 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-03-26T02:22:29.686 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T02:22:29.686 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T02:22:29.731 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:22:31.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:31 smithi164 ceph-mon[29512]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:31.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:31 smithi164 ceph-mon[24851]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:31.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:31 smithi172 ceph-mon[25688]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:33.209 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:33 smithi172 ceph-mon[25688]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:33.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:33 smithi164 ceph-mon[29512]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:33.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:33 smithi164 ceph-mon[24851]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:33.619 INFO:teuthology.orchestra.run.smithi172.stdout:[client.1] 2024-03-26T02:22:33.619 INFO:teuthology.orchestra.run.smithi172.stdout: key = AQBpMQJmbVaoJBAAn8ctRb/xrNdzCQWpLZIikw== 2024-03-26T02:22:34.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:34 smithi164 ceph-mon[29512]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:34.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:34 smithi164 ceph-mon[29512]: from='client.? 172.21.15.172:0/2129788800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:34.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:34 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:34.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:34 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:22:34.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:34 smithi164 ceph-mon[24851]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:34.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:34 smithi164 ceph-mon[24851]: from='client.? 172.21.15.172:0/2129788800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:34.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:34 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:34.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:34 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:22:34.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:34 smithi172 ceph-mon[25688]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:34.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:34 smithi172 ceph-mon[25688]: from='client.? 172.21.15.172:0/2129788800' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:34.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:34 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:22:34.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:34 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:22:34.580 DEBUG:teuthology.orchestra.run.smithi172:> set -ex 2024-03-26T02:22:34.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T02:22:34.581 DEBUG:teuthology.orchestra.run.smithi172:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T02:22:34.623 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T02:22:34.623 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T02:22:34.623 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T02:22:34.845 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:36.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:36 smithi164 ceph-mon[24851]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:36.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:36 smithi164 ceph-mon[29512]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:36.715 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:36 smithi172 ceph-mon[25688]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:36.749 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:22:37.307 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 systemd[1]: Starting Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:22:37.411 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":19,"active_gid":14421,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":2071275137},{"type":"v1","addr":"172.21.15.164:6801","nonce":2071275137}]},"active_addr":"172.21.15.164:6801/2071275137","active_change":"2024-03-26T02:22:19.094707+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34337,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.164:8443/","prometheus":"http://172.21.15.164:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":258825764}]},{"addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":778903593}]},{"addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":3783252420}]},{"addrvec":[{"type":"v2","addr":"172.21.15.164:0","nonce":3230998345}]}]}} 2024-03-26T02:22:37.414 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T02:22:37.415 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T02:22:37.415 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:22:37.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1742344842' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:22:37.586 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 podman[40833]: 2024-03-26 02:22:37.289553185 +0000 UTC m=+0.019184962 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T02:22:37.586 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 podman[40833]: 2024-03-26 02:22:37.39154182 +0000 UTC m=+0.121173592 container create 60410e1fd3de3250d161058eb833eeb94b708bc6a714127906ce90c6ba37d8b1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:22:37.586 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 podman[40833]: 2024-03-26 02:22:37.585494711 +0000 UTC m=+0.315126483 container init 60410e1fd3de3250d161058eb833eeb94b708bc6a714127906ce90c6ba37d8b1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:22:37.635 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:37.674 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:37 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1742344842' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:22:37.674 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:37 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1742344842' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:22:38.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 podman[40833]: 2024-03-26 02:22:37.588526875 +0000 UTC m=+0.318158654 container start 60410e1fd3de3250d161058eb833eeb94b708bc6a714127906ce90c6ba37d8b1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:22:38.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.611Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T02:22:38.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.611Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T02:22:38.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.611Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.611Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi172 (none))" 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.611Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.611Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.612Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.613Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.614Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.616Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.616Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.785µs 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.616Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.616Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.616Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.088µs wal_replay_duration=180.462µs total_replay_duration=214.698µs 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.617Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:22:38.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.617Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T02:22:38.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.617Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:22:38.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.633Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.412826ms db_storage=610ns remote_storage=1.106µs web_handler=260ns query_engine=522ns scrape=478.582µs scrape_sd=16.46µs notify=816ns notify_sd=1.272µs rules=15.755493ms 2024-03-26T02:22:38.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:22:37.633Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T02:22:38.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 bash[40833]: 60410e1fd3de3250d161058eb833eeb94b708bc6a714127906ce90c6ba37d8b1 2024-03-26T02:22:38.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:22:37 smithi172 systemd[1]: Started Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:22:38.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:38 smithi164 ceph-mon[24851]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:38.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:38 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:38.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:38 smithi164 ceph-mon[24851]: Deploying daemon node-exporter.a on smithi164 2024-03-26T02:22:38.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:38 smithi164 ceph-mon[29512]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:38.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:38 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:38.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:38 smithi164 ceph-mon[29512]: Deploying daemon node-exporter.a on smithi164 2024-03-26T02:22:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:38 smithi172 ceph-mon[25688]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:38 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:38 smithi172 ceph-mon[25688]: Deploying daemon node-exporter.a on smithi164 2024-03-26T02:22:39.436 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:22:39.436 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":59,"fsid":"e2654d3c-eb16-11ee-95d0-87774f69a715","created":"2024-03-26T02:17:14.051039+0000","modified":"2024-03-26T02:22:19.094341+0000","last_up_change":"2024-03-26T02:22:03.237622+0000","last_in_change":"2024-03-26T02:21:46.121780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:20:07.678670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fa926af6-17d0-4176-b716-b668e463738d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6803","nonce":378049558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6805","nonce":378049558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6809","nonce":378049558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6807","nonce":378049558}]},"public_addr":"172.21.15.164:6803/378049558","cluster_addr":"172.21.15.164:6805/378049558","heartbeat_back_addr":"172.21.15.164:6809/378049558","heartbeat_front_addr":"172.21.15.164:6807/378049558","state":["exists","up"]},{"osd":1,"uuid":"662f7218-b733-4bd5-9c05-305d11ec6b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6811","nonce":3337400680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6813","nonce":3337400680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6817","nonce":3337400680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6815","nonce":3337400680}]},"public_addr":"172.21.15.164:6811/3337400680","cluster_addr":"172.21.15.164:6813/3337400680","heartbeat_back_addr":"172.21.15.164:6817/3337400680","heartbeat_front_addr":"172.21.15.164:6815/3337400680","state":["exists","up"]},{"osd":2,"uuid":"cea9695c-f306-468a-8071-debc4326b99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6819","nonce":851694553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6821","nonce":851694553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6825","nonce":851694553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6823","nonce":851694553}]},"public_addr":"172.21.15.164:6819/851694553","cluster_addr":"172.21.15.164:6821/851694553","heartbeat_back_addr":"172.21.15.164:6825/851694553","heartbeat_front_addr":"172.21.15.164:6823/851694553","state":["exists","up"]},{"osd":3,"uuid":"b90e1e0c-0eee-46e3-b56c-54e8aea3bb75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6827","nonce":3222350232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6829","nonce":3222350232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6833","nonce":3222350232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6831","nonce":3222350232}]},"public_addr":"172.21.15.164:6827/3222350232","cluster_addr":"172.21.15.164:6829/3222350232","heartbeat_back_addr":"172.21.15.164:6833/3222350232","heartbeat_front_addr":"172.21.15.164:6831/3222350232","state":["exists","up"]},{"osd":4,"uuid":"f73412ca-7838-48e2-a5dc-897dccf6b4bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6801","nonce":3407051165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6803","nonce":3407051165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6807","nonce":3407051165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6805","nonce":3407051165}]},"public_addr":"172.21.15.172:6801/3407051165","cluster_addr":"172.21.15.172:6803/3407051165","heartbeat_back_addr":"172.21.15.172:6807/3407051165","heartbeat_front_addr":"172.21.15.172:6805/3407051165","state":["exists","up"]},{"osd":5,"uuid":"deea163e-0adc-4ae4-a720-e0758a614f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6809","nonce":887601481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6811","nonce":887601481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6815","nonce":887601481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6813","nonce":887601481}]},"public_addr":"172.21.15.172:6809/887601481","cluster_addr":"172.21.15.172:6811/887601481","heartbeat_back_addr":"172.21.15.172:6815/887601481","heartbeat_front_addr":"172.21.15.172:6813/887601481","state":["exists","up"]},{"osd":6,"uuid":"83914243-cedc-4cfe-9b0b-4dfc37d72d23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6817","nonce":928726954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6819","nonce":928726954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6823","nonce":928726954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6821","nonce":928726954}]},"public_addr":"172.21.15.172:6817/928726954","cluster_addr":"172.21.15.172:6819/928726954","heartbeat_back_addr":"172.21.15.172:6823/928726954","heartbeat_front_addr":"172.21.15.172:6821/928726954","state":["exists","up"]},{"osd":7,"uuid":"fcd7fb7d-3b97-4337-8264-8312679cd9d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6825","nonce":3771214110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6827","nonce":3771214110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6831","nonce":3771214110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6829","nonce":3771214110}]},"public_addr":"172.21.15.172:6825/3771214110","cluster_addr":"172.21.15.172:6827/3771214110","heartbeat_back_addr":"172.21.15.172:6831/3771214110","heartbeat_front_addr":"172.21.15.172:6829/3771214110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:19:17.847711+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:19:41.445819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:04.505257+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:27.790127+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:49.467227+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:21:13.368554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:21:37.449411+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:22:02.106264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:0/78250723":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/3236129366":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/2490859237":"2024-03-27T02:22:19.094293+0000","172.21.15.164:6800/1272665921":"2024-03-27T02:17:59.295370+0000","172.21.15.164:6800/2200155095":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/1798451206":"2024-03-27T02:17:59.295370+0000","172.21.15.164:0/3009346081":"2024-03-27T02:17:59.295370+0000","172.21.15.164:0/3498887725":"2024-03-27T02:17:37.995264+0000","172.21.15.164:0/711619639":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/1487529581":"2024-03-27T02:17:37.995264+0000","172.21.15.164:0/3486476943":"2024-03-27T02:17:59.295370+0000","172.21.15.164:6801/1272665921":"2024-03-27T02:17:59.295370+0000","172.21.15.164:6801/2862881695":"2024-03-27T02:17:37.995264+0000","172.21.15.164:6800/2862881695":"2024-03-27T02:17:37.995264+0000","172.21.15.164:6801/2200155095":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/75066052":"2024-03-27T02:17:37.995264+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:22:40.298 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T02:22:40.298 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:22:40.529 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:40.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:40 smithi164 ceph-mon[24851]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:40.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:40 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:40.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:40 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/276267059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:40.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:40 smithi164 ceph-mon[29512]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:40.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:40 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:40.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:40 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/276267059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:40.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:40 smithi164 bash[44472]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-26T02:22:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:40 smithi172 ceph-mon[25688]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:40 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:40 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/276267059' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:40.957 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:40 smithi164 bash[44472]: Getting image source signatures 2024-03-26T02:22:40.957 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:40 smithi164 bash[44472]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-26T02:22:40.957 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:40 smithi164 bash[44472]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-26T02:22:40.957 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:40 smithi164 bash[44472]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-26T02:22:42.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:42 smithi164 ceph-mon[24851]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:42.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:42 smithi164 ceph-mon[29512]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:42.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:42 smithi172 ceph-mon[25688]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:42.856 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:42 smithi164 bash[44472]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-26T02:22:43.179 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:42 smithi164 bash[44472]: Writing manifest to image destination 2024-03-26T02:22:43.313 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:22:43.313 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":59,"fsid":"e2654d3c-eb16-11ee-95d0-87774f69a715","created":"2024-03-26T02:17:14.051039+0000","modified":"2024-03-26T02:22:19.094341+0000","last_up_change":"2024-03-26T02:22:03.237622+0000","last_in_change":"2024-03-26T02:21:46.121780+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T02:20:07.678670+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"fa926af6-17d0-4176-b716-b668e463738d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6803","nonce":378049558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6805","nonce":378049558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6809","nonce":378049558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":378049558},{"type":"v1","addr":"172.21.15.164:6807","nonce":378049558}]},"public_addr":"172.21.15.164:6803/378049558","cluster_addr":"172.21.15.164:6805/378049558","heartbeat_back_addr":"172.21.15.164:6809/378049558","heartbeat_front_addr":"172.21.15.164:6807/378049558","state":["exists","up"]},{"osd":1,"uuid":"662f7218-b733-4bd5-9c05-305d11ec6b26","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6811","nonce":3337400680}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6813","nonce":3337400680}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6817","nonce":3337400680}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":3337400680},{"type":"v1","addr":"172.21.15.164:6815","nonce":3337400680}]},"public_addr":"172.21.15.164:6811/3337400680","cluster_addr":"172.21.15.164:6813/3337400680","heartbeat_back_addr":"172.21.15.164:6817/3337400680","heartbeat_front_addr":"172.21.15.164:6815/3337400680","state":["exists","up"]},{"osd":2,"uuid":"cea9695c-f306-468a-8071-debc4326b99b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6819","nonce":851694553}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6821","nonce":851694553}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6824","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6825","nonce":851694553}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":851694553},{"type":"v1","addr":"172.21.15.164:6823","nonce":851694553}]},"public_addr":"172.21.15.164:6819/851694553","cluster_addr":"172.21.15.164:6821/851694553","heartbeat_back_addr":"172.21.15.164:6825/851694553","heartbeat_front_addr":"172.21.15.164:6823/851694553","state":["exists","up"]},{"osd":3,"uuid":"b90e1e0c-0eee-46e3-b56c-54e8aea3bb75","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6826","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6827","nonce":3222350232}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6828","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6829","nonce":3222350232}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6832","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6833","nonce":3222350232}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6830","nonce":3222350232},{"type":"v1","addr":"172.21.15.164:6831","nonce":3222350232}]},"public_addr":"172.21.15.164:6827/3222350232","cluster_addr":"172.21.15.164:6829/3222350232","heartbeat_back_addr":"172.21.15.164:6833/3222350232","heartbeat_front_addr":"172.21.15.164:6831/3222350232","state":["exists","up"]},{"osd":4,"uuid":"f73412ca-7838-48e2-a5dc-897dccf6b4bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6800","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6801","nonce":3407051165}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6802","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6803","nonce":3407051165}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6806","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6807","nonce":3407051165}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6804","nonce":3407051165},{"type":"v1","addr":"172.21.15.172:6805","nonce":3407051165}]},"public_addr":"172.21.15.172:6801/3407051165","cluster_addr":"172.21.15.172:6803/3407051165","heartbeat_back_addr":"172.21.15.172:6807/3407051165","heartbeat_front_addr":"172.21.15.172:6805/3407051165","state":["exists","up"]},{"osd":5,"uuid":"deea163e-0adc-4ae4-a720-e0758a614f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6808","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6809","nonce":887601481}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6810","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6811","nonce":887601481}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6814","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6815","nonce":887601481}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6812","nonce":887601481},{"type":"v1","addr":"172.21.15.172:6813","nonce":887601481}]},"public_addr":"172.21.15.172:6809/887601481","cluster_addr":"172.21.15.172:6811/887601481","heartbeat_back_addr":"172.21.15.172:6815/887601481","heartbeat_front_addr":"172.21.15.172:6813/887601481","state":["exists","up"]},{"osd":6,"uuid":"83914243-cedc-4cfe-9b0b-4dfc37d72d23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6816","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6817","nonce":928726954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6818","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6819","nonce":928726954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6822","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6823","nonce":928726954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6820","nonce":928726954},{"type":"v1","addr":"172.21.15.172:6821","nonce":928726954}]},"public_addr":"172.21.15.172:6817/928726954","cluster_addr":"172.21.15.172:6819/928726954","heartbeat_back_addr":"172.21.15.172:6823/928726954","heartbeat_front_addr":"172.21.15.172:6821/928726954","state":["exists","up"]},{"osd":7,"uuid":"fcd7fb7d-3b97-4337-8264-8312679cd9d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6824","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6825","nonce":3771214110}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6826","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6827","nonce":3771214110}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6830","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6831","nonce":3771214110}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.172:6828","nonce":3771214110},{"type":"v1","addr":"172.21.15.172:6829","nonce":3771214110}]},"public_addr":"172.21.15.172:6825/3771214110","cluster_addr":"172.21.15.172:6827/3771214110","heartbeat_back_addr":"172.21.15.172:6831/3771214110","heartbeat_front_addr":"172.21.15.172:6829/3771214110","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:19:17.847711+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:19:41.445819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:04.505257+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:27.790127+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:20:49.467227+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:21:13.368554+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:21:37.449411+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:22:02.106264+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.164:0/78250723":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/3236129366":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/2490859237":"2024-03-27T02:22:19.094293+0000","172.21.15.164:6800/1272665921":"2024-03-27T02:17:59.295370+0000","172.21.15.164:6800/2200155095":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/1798451206":"2024-03-27T02:17:59.295370+0000","172.21.15.164:0/3009346081":"2024-03-27T02:17:59.295370+0000","172.21.15.164:0/3498887725":"2024-03-27T02:17:37.995264+0000","172.21.15.164:0/711619639":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/1487529581":"2024-03-27T02:17:37.995264+0000","172.21.15.164:0/3486476943":"2024-03-27T02:17:59.295370+0000","172.21.15.164:6801/1272665921":"2024-03-27T02:17:59.295370+0000","172.21.15.164:6801/2862881695":"2024-03-27T02:17:37.995264+0000","172.21.15.164:6800/2862881695":"2024-03-27T02:17:37.995264+0000","172.21.15.164:6801/2200155095":"2024-03-27T02:22:19.094293+0000","172.21.15.164:0/75066052":"2024-03-27T02:17:37.995264+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:22:43.529 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 podman[44472]: 2024-03-26 02:22:43.307165035 +0000 UTC m=+2.947560718 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-26T02:22:43.529 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 podman[44472]: 2024-03-26 02:22:43.423098596 +0000 UTC m=+3.063494278 container create 6fbe35cf8a99cb29336eda977758491b5a5d206276bc07c3e8f50e990aa05bf9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 podman[44472]: 2024-03-26 02:22:43.658713382 +0000 UTC m=+3.299109074 container init 6fbe35cf8a99cb29336eda977758491b5a5d206276bc07c3e8f50e990aa05bf9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 podman[44472]: 2024-03-26 02:22:43.661605624 +0000 UTC m=+3.302001308 container start 6fbe35cf8a99cb29336eda977758491b5a5d206276bc07c3e8f50e990aa05bf9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.665Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.665Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.665Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.665Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=arp 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-26T02:22:43.795 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-26T02:22:43.796 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-26T02:22:43.796 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-26T02:22:43.796 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-26T02:22:43.796 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-26T02:22:43.796 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-26T02:22:43.796 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=edac 2024-03-26T02:22:43.796 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-26T02:22:43.797 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=os 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=stat 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=time 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-26T02:22:43.798 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=uname 2024-03-26T02:22:43.799 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-26T02:22:43.799 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-26T02:22:43.799 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-26T02:22:43.799 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.666Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-26T02:22:43.799 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[44647]: ts=2024-03-26T02:22:43.667Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-26T02:22:43.799 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 bash[44472]: 6fbe35cf8a99cb29336eda977758491b5a5d206276bc07c3e8f50e990aa05bf9 2024-03-26T02:22:43.799 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:22:43 smithi164 systemd[1]: Started Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:22:44.445 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:44 smithi172 ceph-mon[25688]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:44.445 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:44 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1338558395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:44.445 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:44 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:44.445 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:44 smithi172 ceph-mon[25688]: Deploying daemon node-exporter.b on smithi172 2024-03-26T02:22:44.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:44 smithi164 ceph-mon[24851]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:44.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:44 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1338558395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:44.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:44 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:44.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:44 smithi164 ceph-mon[24851]: Deploying daemon node-exporter.b on smithi172 2024-03-26T02:22:44.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:44 smithi164 ceph-mon[29512]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:44.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:44 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1338558395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:22:44.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:44 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:44.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:44 smithi164 ceph-mon[29512]: Deploying daemon node-exporter.b on smithi172 2024-03-26T02:22:44.586 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T02:22:44.586 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T02:22:44.587 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T02:22:44.587 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T02:22:44.587 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T02:22:44.588 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T02:22:44.588 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T02:22:44.589 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T02:22:44.807 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:44 smithi172 bash[41071]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-03-26T02:22:45.058 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:45.058 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:45.059 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:45.059 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:45.061 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:45.061 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:45.061 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:22:45.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:45 smithi172 bash[41071]: Getting image source signatures 2024-03-26T02:22:45.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:45 smithi172 bash[41071]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-03-26T02:22:45.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:45 smithi172 bash[41071]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-03-26T02:22:45.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:45 smithi172 bash[41071]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-03-26T02:22:46.646 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:46 smithi164 ceph-mon[29512]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:46.646 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:46 smithi164 ceph-mon[24851]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:46 smithi172 ceph-mon[25688]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:48.307 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:47 smithi172 bash[41071]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-03-26T02:22:48.307 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 bash[41071]: Writing manifest to image destination 2024-03-26T02:22:48.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:22:48.618 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-mon[25688]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:48.618 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 podman[41071]: 2024-03-26 02:22:48.275187372 +0000 UTC m=+3.663806790 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-26T02:22:48.618 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 podman[41071]: 2024-03-26 02:22:48.407348599 +0000 UTC m=+3.795968007 container create 9c5963ac47a19945674c24a86f0c42ff0d5da17d6694da43d3ddf29f8aff3fc7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:22:48.671 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:48 smithi164 ceph-mon[24851]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:48.672 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:48 smithi164 ceph-mon[29512]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 podman[41071]: 2024-03-26 02:22:48.617782611 +0000 UTC m=+4.006402017 container init 9c5963ac47a19945674c24a86f0c42ff0d5da17d6694da43d3ddf29f8aff3fc7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 podman[41071]: 2024-03-26 02:22:48.621532937 +0000 UTC m=+4.010152352 container start 9c5963ac47a19945674c24a86f0c42ff0d5da17d6694da43d3ddf29f8aff3fc7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.625Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.625Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.625Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.625Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=arp 2024-03-26T02:22:48.940 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=edac 2024-03-26T02:22:48.941 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-26T02:22:48.942 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=os 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=stat 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-26T02:22:48.943 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=time 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=uname 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[41122]: ts=2024-03-26T02:22:48.626Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 bash[41071]: 9c5963ac47a19945674c24a86f0c42ff0d5da17d6694da43d3ddf29f8aff3fc7 2024-03-26T02:22:48.944 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:22:48 smithi172 systemd[1]: Started Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:22:50.015 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:49 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:50.015 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:49 smithi164 ceph-mon[24851]: Deploying daemon alertmanager.a on smithi164 2024-03-26T02:22:50.015 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:49 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:50.015 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:49 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:50.016 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:49 smithi164 ceph-mon[29512]: Deploying daemon alertmanager.a on smithi164 2024-03-26T02:22:50.016 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:49 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:49 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:49 smithi172 ceph-mon[25688]: Deploying daemon alertmanager.a on smithi164 2024-03-26T02:22:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:49 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:22:51.028 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:50 smithi164 ceph-mon[24851]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:51.028 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:50 smithi164 ceph-mon[29512]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:50 smithi172 ceph-mon[25688]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:52.132 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:22:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:22:51] "GET /metrics HTTP/1.1" 200 191491 "" "Prometheus/2.33.4" 2024-03-26T02:22:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:52 smithi164 ceph-mon[24851]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:53.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:52 smithi164 ceph-mon[29512]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:52 smithi172 ceph-mon[25688]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:54.633 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:54 smithi164 ceph-mon[24851]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:54.633 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:54 smithi164 ceph-mon[29512]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:54.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:54 smithi172 ceph-mon[25688]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:56.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:56 smithi164 ceph-mon[24851]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:56.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:56 smithi164 ceph-mon[29512]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:56 smithi172 ceph-mon[25688]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:57.689 INFO:teuthology.orchestra.run.smithi164.stdout:55834574888 2024-03-26T02:22:57.689 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T02:22:58.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:22:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:22:59.032 INFO:teuthology.orchestra.run.smithi164.stdout:231928233996 2024-03-26T02:22:59.033 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T02:22:59.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:22:59 smithi164 ceph-mon[24851]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:59.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:22:59 smithi164 ceph-mon[29512]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:22:59.434 INFO:teuthology.orchestra.run.smithi164.stdout:34359738413 2024-03-26T02:22:59.435 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T02:22:59.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:22:59 smithi172 ceph-mon[25688]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:01.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:01 smithi164 ceph-mon[24851]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:01.527 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:01 smithi164 ceph-mon[29512]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:01.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:01 smithi172 ceph-mon[25688]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:01.961 INFO:teuthology.orchestra.run.smithi164.stdout:77309411364 2024-03-26T02:23:01.962 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T02:23:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:23:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:01] "GET /metrics HTTP/1.1" 200 191491 "" "Prometheus/2.33.4" 2024-03-26T02:23:02.427 INFO:teuthology.orchestra.run.smithi164.stdout:163208757268 2024-03-26T02:23:02.427 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T02:23:02.947 INFO:teuthology.orchestra.run.smithi164.stdout:107374182432 2024-03-26T02:23:02.947 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T02:23:03.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:03 smithi164 ceph-mon[24851]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:03.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:03 smithi164 ceph-mon[29512]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:03.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:03 smithi172 ceph-mon[25688]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:04.071 INFO:teuthology.orchestra.run.smithi164.stdout:197568495632 2024-03-26T02:23:04.071 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T02:23:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:05.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:05 smithi164 ceph-mon[24851]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:05.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:05 smithi164 ceph-mon[29512]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:05.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:05 smithi172 ceph-mon[25688]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:06.758 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:06 smithi164 ceph-mon[24851]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:06.758 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:06 smithi164 ceph-mon[29512]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:06.786 INFO:teuthology.orchestra.run.smithi164.stdout:128849018908 2024-03-26T02:23:06.787 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T02:23:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:06 smithi172 ceph-mon[25688]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:06.818 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:06.818 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:07.399 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:08.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:23:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:23:09.432 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:09.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:09 smithi164 ceph-mon[24851]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:09.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:09 smithi164 ceph-mon[29512]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:09.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:09 smithi172 ceph-mon[25688]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:09.603 INFO:teuthology.orchestra.run.smithi164.stdout:34359738415 2024-03-26T02:23:10.371 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738415 for osd.0 2024-03-26T02:23:10.371 DEBUG:teuthology.parallel:result is None 2024-03-26T02:23:10.515 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:10 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3409025613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:23:10.515 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:10 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3409025613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:23:10.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:10 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3409025613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:23:11.043 INFO:teuthology.orchestra.run.smithi164.stdout:197568495635 2024-03-26T02:23:11.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:11 smithi164 ceph-mon[24851]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:11.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:11 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/206227772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:23:11.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:11 smithi164 ceph-mon[29512]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:11.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:11 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/206227772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:23:11.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:11 smithi172 ceph-mon[25688]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:11.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:11 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/206227772' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:23:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:23:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:11] "GET /metrics HTTP/1.1" 200 191489 "" "Prometheus/2.33.4" 2024-03-26T02:23:12.559 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495632 got 197568495635 for osd.6 2024-03-26T02:23:12.559 DEBUG:teuthology.parallel:result is None 2024-03-26T02:23:12.670 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:12 smithi164 ceph-mon[29512]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:12.671 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:12 smithi164 ceph-mon[24851]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:12.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:12 smithi172 ceph-mon[25688]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:15.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:15 smithi164 ceph-mon[24851]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:15.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:15 smithi164 ceph-mon[29512]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:15.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:15 smithi172 ceph-mon[25688]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:17.390 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:17 smithi164 ceph-mon[24851]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:17.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:17 smithi164 ceph-mon[29512]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:17.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:17 smithi172 ceph-mon[25688]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:17.817 INFO:teuthology.orchestra.run.smithi164.stdout:163208757273 2024-03-26T02:23:18.265 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:23:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:23:18.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:18 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1403365320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T02:23:18.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:18 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1403365320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T02:23:18.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:18 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1403365320' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T02:23:18.794 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757268 got 163208757273 for osd.5 2024-03-26T02:23:18.794 DEBUG:teuthology.parallel:result is None 2024-03-26T02:23:19.391 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:19 smithi164 ceph-mon[24851]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:19.391 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:19 smithi164 ceph-mon[29512]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:19.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:19 smithi172 ceph-mon[25688]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:19.793 INFO:teuthology.orchestra.run.smithi164.stdout:128849018911 2024-03-26T02:23:20.414 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:20 smithi164 ceph-mon[24851]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:20.414 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:23:20.414 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:23:20.414 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:20 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2937791978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T02:23:20.415 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:20 smithi164 ceph-mon[29512]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:20.415 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:23:20.415 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:23:20.415 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:20 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2937791978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T02:23:20.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:20 smithi172 ceph-mon[25688]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:20.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:23:20.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:23:20.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:20 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2937791978' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T02:23:20.649 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018911 for osd.4 2024-03-26T02:23:20.649 DEBUG:teuthology.parallel:result is None 2024-03-26T02:23:21.374 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:21 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2609503823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:23:21.374 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:21 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2609503823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:23:21.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:21 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2609503823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:23:21.767 INFO:teuthology.orchestra.run.smithi164.stdout:55834574894 2024-03-26T02:23:21.843 INFO:teuthology.orchestra.run.smithi164.stdout:77309411369 2024-03-26T02:23:22.272 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:23:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:21] "GET /metrics HTTP/1.1" 200 191488 "" "Prometheus/2.33.4" 2024-03-26T02:23:22.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[29512]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:22.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1545313307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[24851]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1545313307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:22 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:22 smithi172 ceph-mon[25688]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:22 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1545313307' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:23:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:22 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:22 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:22 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:22 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:22.706 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411369 for osd.2 2024-03-26T02:23:22.707 DEBUG:teuthology.parallel:result is None 2024-03-26T02:23:23.446 INFO:teuthology.orchestra.run.smithi164.stdout:107374182436 2024-03-26T02:23:23.622 INFO:teuthology.orchestra.run.smithi164.stdout:231928234001 2024-03-26T02:23:23.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[47099]: level=info ts=2024-03-26T02:23:23.668Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000120931s 2024-03-26T02:23:24.315 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233996 got 231928234001 for osd.7 2024-03-26T02:23:24.316 DEBUG:teuthology.parallel:result is None 2024-03-26T02:23:24.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:24 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1812823703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T02:23:24.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:24 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1118594219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T02:23:24.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:24 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1812823703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T02:23:24.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:24 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1118594219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T02:23:24.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:24 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1812823703' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T02:23:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:24 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1118594219' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T02:23:24.710 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182432 got 107374182436 for osd.3 2024-03-26T02:23:24.710 DEBUG:teuthology.parallel:result is None 2024-03-26T02:23:25.156 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574894 for osd.1 2024-03-26T02:23:25.156 DEBUG:teuthology.parallel:result is None 2024-03-26T02:23:25.156 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T02:23:25.156 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T02:23:25.491 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:25 smithi164 ceph-mon[29512]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:25.491 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:25 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:25.491 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:25 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:25.491 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:25 smithi164 ceph-mon[24851]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:25.491 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:25 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:25.492 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:25 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:25.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:25 smithi172 ceph-mon[25688]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:25.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:25 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:25.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:25 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:25.834 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:26.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:26 smithi164 ceph-mon[24851]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:26.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:26 smithi164 ceph-mon[29512]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:26.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:26 smithi172 ceph-mon[25688]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:27.671 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:23:27.673 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-03-26T02:23:28.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:23:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:23:28.868 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-03-26T02:23:27.266040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6084,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6230016,"data_stored":3488013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002128"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-26T02:22:19.337695+0000","last_change":"2024-03-26T02:22:07.268464+0000","last_active":"2024-03-26T02:22:19.337695+0000","last_peered":"2024-03-26T02:22:19.337695+0000","last_clean":"2024-03-26T02:22:19.337695+0000","last_became_active":"2024-03-26T02:22:07.268052+0000","last_became_peered":"2024-03-26T02:22:07.268052+0000","last_unstale":"2024-03-26T02:22:19.337695+0000","last_undegraded":"2024-03-26T02:22:19.337695+0000","last_fullsized":"2024-03-26T02:22:19.337695+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:20:07.679132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:20:07.679132+0000","last_clean_scrub_stamp":"2024-03-26T02:20:07.679132+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:27:27.957156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6080,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739072,"statfs":{"total":95995035648,"available":95988809728,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.439},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.61099999999999999}]},{"osd":1,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.54800000000000004}]},{"osd":2,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.496},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.57299999999999995}]},{"osd":3,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.52900000000000003}]},{"osd":4,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.51200000000000001}]},{"osd":5,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.59699999999999998}]},{"osd":6,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.66900000000000004}]}]},{"osd":6,"up_from":46,"seq":197568495639,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.39200000000000002}]},{"osd":1,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.46200000000000002}]},{"osd":3,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.63300000000000001}]},{"osd":4,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.48499999999999999}]},{"osd":5,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.54800000000000004}]},{"osd":7,"last update":"Tue Mar 26 02:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65800000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.33800000000000002},"min":{"1min":0.27600000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.373,"15min":0.35699999999999998},"min":{"1min":0.28599999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.30199999999999999}]},{"osd":2,"last update":"Tue Mar 26 02:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59999999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39300000000000002,"15min":0.38300000000000001},"min":{"1min":0.309,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.36299999999999999}]},{"osd":3,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42499999999999999,"15min":0.42699999999999999},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.51500000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.35899999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.376}]},{"osd":4,"last update":"Tue Mar 26 02:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.46400000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.42799999999999999}]},{"osd":5,"last update":"Tue Mar 26 02:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.45600000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.752,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52400000000000002}]},{"osd":6,"last update":"Tue Mar 26 02:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.502},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.44900000000000001}]},{"osd":7,"last update":"Tue Mar 26 02:23:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.442},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.51500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 26 02:22:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.34200000000000003,"15min":0.315},"min":{"1min":0.255,"5min":0.214,"15min":0.214},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.318},"min":{"1min":0.29999999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.32600000000000001}]},{"osd":2,"last update":"Tue Mar 26 02:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.34699999999999998},"min":{"1min":0.28999999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.52500000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.38800000000000001,"15min":0.36799999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.58199999999999996,"5min":0.621,"15min":0.621},"last":0.35899999999999999}]},{"osd":3,"last update":"Tue Mar 26 02:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.43099999999999999,"15min":0.42499999999999999},"min":{"1min":0.32100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.442,"15min":0.434},"min":{"1min":0.379,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Mar 26 02:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.504,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73899999999999999}]},{"osd":5,"last update":"Tue Mar 26 02:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Mar 26 02:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.622}]},{"osd":7,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.68799999999999994}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54200000000000004,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.36599999999999999},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.50600000000000001,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.29799999999999999}]},{"osd":1,"last update":"Tue Mar 26 02:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.35399999999999998},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.39100000000000001,"15min":0.375},"min":{"1min":0.252,"5min":0.247,"15min":0.247},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.32200000000000001}]},{"osd":3,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.33500000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56000000000000005,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.314}]},{"osd":4,"last update":"Tue Mar 26 02:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.71399999999999997}]},{"osd":5,"last update":"Tue Mar 26 02:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57499999999999996,"15min":0.57399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41599999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.56200000000000006}]},{"osd":6,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.71899999999999997}]},{"osd":7,"last update":"Tue Mar 26 02:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.58599999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182437,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50700000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.56699999999999995}]},{"osd":1,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.48899999999999999}]},{"osd":2,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60999999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.438,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59899999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.40600000000000003}]},{"osd":4,"last update":"Tue Mar 26 02:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78400000000000003,"5min":0.878,"15min":0.878},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80000000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.54700000000000004}]},{"osd":5,"last update":"Tue Mar 26 02:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.82599999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52600000000000002}]},{"osd":6,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.751}]},{"osd":7,"last update":"Tue Mar 26 02:23:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.62}]}]},{"osd":4,"up_from":30,"seq":128849018913,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.65100000000000002,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.54200000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.745,"15min":0.745},"last":0.41099999999999998}]},{"osd":1,"last update":"Tue Mar 26 02:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.41399999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.751,"5min":0.753,"15min":0.753},"last":0.372},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.70699999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.497}]},{"osd":2,"last update":"Tue Mar 26 02:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.439},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.503}]},{"osd":3,"last update":"Tue Mar 26 02:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55600000000000005,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46500000000000002}]},{"osd":5,"last update":"Tue Mar 26 02:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.375},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.318},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.39900000000000002,"15min":0.38},"min":{"1min":0.30599999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.44500000000000001}]},{"osd":6,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.47699999999999998}]},{"osd":7,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.43099999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.42799999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.52700000000000002}]},{"osd":1,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56699999999999995,"15min":0.56599999999999995},"min":{"1min":0.41299999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.69399999999999995,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.39300000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69799999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.48299999999999998}]},{"osd":2,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.69599999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.46899999999999997}]},{"osd":3,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.47899999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.68500000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.69399999999999995,"5min":0.755,"15min":0.755},"last":0.51200000000000001}]},{"osd":4,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.41499999999999998},"min":{"1min":0.33200000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.42899999999999999,"15min":0.42099999999999999},"min":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39400000000000002}]},{"osd":6,"last update":"Tue Mar 26 02:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.436},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.54900000000000004}]},{"osd":7,"last update":"Tue Mar 26 02:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.42299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T02:23:28.870 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T02:23:29.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:29 smithi164 ceph-mon[24851]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:29.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:29 smithi164 ceph-mon[24851]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:23:29.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:29 smithi164 ceph-mon[29512]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:29.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:29 smithi164 ceph-mon[29512]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:23:29.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:29 smithi172 ceph-mon[25688]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:29.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:29 smithi172 ceph-mon[25688]: from='client.24505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:23:29.623 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:30.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:30.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.891 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[29512]: Deploying daemon grafana.a on smithi172 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:30.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:30 smithi164 ceph-mon[24851]: Deploying daemon grafana.a on smithi172 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:30 smithi172 ceph-mon[25688]: Deploying daemon grafana.a on smithi172 2024-03-26T02:23:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:23:31.411 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-03-26T02:23:31.967 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[47099]: level=info ts=2024-03-26T02:23:31.670Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002340528s 2024-03-26T02:23:32.056 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-03-26T02:23:31.266716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48900,"kb_used_data":6084,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912316,"statfs":{"total":767960285184,"available":767910211584,"internally_reserved":0,"allocated":6230016,"data_stored":3488013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002091"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-03-26T02:22:19.337695+0000","last_change":"2024-03-26T02:22:07.268464+0000","last_active":"2024-03-26T02:22:19.337695+0000","last_peered":"2024-03-26T02:22:19.337695+0000","last_clean":"2024-03-26T02:22:19.337695+0000","last_became_active":"2024-03-26T02:22:07.268052+0000","last_became_peered":"2024-03-26T02:22:07.268052+0000","last_unstale":"2024-03-26T02:22:19.337695+0000","last_undegraded":"2024-03-26T02:22:19.337695+0000","last_fullsized":"2024-03-26T02:22:19.337695+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:20:07.679132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:20:07.679132+0000","last_clean_scrub_stamp":"2024-03-26T02:20:07.679132+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:27:27.957156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234003,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6080,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739072,"statfs":{"total":95995035648,"available":95988809728,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59499999999999997}]},{"osd":1,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.58299999999999996}]},{"osd":2,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.63400000000000001}]},{"osd":3,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.60899999999999999}]},{"osd":4,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.54000000000000004}]},{"osd":5,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.46600000000000003}]},{"osd":6,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.499}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6144,"kb_used_data":1080,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739008,"statfs":{"total":95995035648,"available":95988744192,"internally_reserved":0,"allocated":1105920,"data_stored":766181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.54600000000000004}]},{"osd":1,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.625},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.59099999999999997}]},{"osd":2,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.57799999999999996}]},{"osd":3,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.63700000000000001}]},{"osd":4,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.56899999999999995}]},{"osd":5,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.437},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.50600000000000001}]},{"osd":7,"last update":"Tue Mar 26 02:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.56200000000000006}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6336,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738816,"statfs":{"total":95995035648,"available":95988547584,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:22:47 2024","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.35599999999999998,"15min":0.33800000000000002},"min":{"1min":0.27600000000000002,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.373,"15min":0.35699999999999998},"min":{"1min":0.28599999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26300000000000001}]},{"osd":2,"last update":"Tue Mar 26 02:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.41899999999999998,"15min":0.41099999999999998},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.59999999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.39300000000000002,"15min":0.38300000000000001},"min":{"1min":0.309,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.36499999999999999}]},{"osd":3,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42499999999999999,"15min":0.42699999999999999},"min":{"1min":0.28799999999999998,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.51500000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.248},{"interface":"front","average":{"1min":0.495,"5min":0.47199999999999998,"15min":0.46800000000000003},"min":{"1min":0.35899999999999999,"5min":0.316,"15min":0.316},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.29699999999999999}]},{"osd":4,"last update":"Tue Mar 26 02:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57199999999999995,"15min":0.56799999999999995},"min":{"1min":0.46400000000000002,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45300000000000001}]},{"osd":5,"last update":"Tue Mar 26 02:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.45600000000000002,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.752,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.46200000000000002}]},{"osd":6,"last update":"Tue Mar 26 02:22:42 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.434}]},{"osd":7,"last update":"Tue Mar 26 02:23:07 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.443},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.46899999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6920,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738232,"statfs":{"total":95995035648,"available":95987949568,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Mar 26 02:22:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.34200000000000003,"15min":0.315},"min":{"1min":0.255,"5min":0.214,"15min":0.214},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.318},"min":{"1min":0.29999999999999999,"5min":0.22,"15min":0.22},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.253}]},{"osd":2,"last update":"Tue Mar 26 02:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.36699999999999999,"15min":0.34699999999999998},"min":{"1min":0.28999999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.52500000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.38800000000000001,"15min":0.36799999999999999},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.58199999999999996,"5min":0.621,"15min":0.621},"last":0.26100000000000001}]},{"osd":3,"last update":"Tue Mar 26 02:22:31 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.43099999999999999,"15min":0.42499999999999999},"min":{"1min":0.32100000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.442,"15min":0.434},"min":{"1min":0.379,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.35799999999999998}]},{"osd":4,"last update":"Tue Mar 26 02:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.58099999999999996,"15min":0.56999999999999995},"min":{"1min":0.504,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.432},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.55800000000000005},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53300000000000003}]},{"osd":5,"last update":"Tue Mar 26 02:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.46999999999999997,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.60599999999999998},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.54300000000000004}]},{"osd":6,"last update":"Tue Mar 26 02:22:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.65200000000000002}]},{"osd":7,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58999999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.36499999999999999,"15min":0.36099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.54200000000000004,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.44},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.371,"15min":0.36599999999999999},"min":{"1min":0.26800000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.50600000000000001,"5min":0.53700000000000003,"15min":0.53700000000000003},"last":0.46700000000000003}]},{"osd":1,"last update":"Tue Mar 26 02:23:11 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.36399999999999999,"15min":0.35399999999999998},"min":{"1min":0.26500000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.51500000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.39100000000000001,"15min":0.375},"min":{"1min":0.252,"5min":0.247,"15min":0.247},"max":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48899999999999999}]},{"osd":3,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.33500000000000002,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.56000000000000005,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.498},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.443,"15min":0.44500000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.47899999999999998}]},{"osd":4,"last update":"Tue Mar 26 02:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.60299999999999998,"15min":0.60999999999999999},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.61399999999999999},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.81699999999999995}]},{"osd":5,"last update":"Tue Mar 26 02:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57499999999999996,"15min":0.57399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"min":{"1min":0.41599999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.877}]},{"osd":6,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.92000000000000004}]},{"osd":7,"last update":"Tue Mar 26 02:23:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.85699999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50700000000000001,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.495}]},{"osd":1,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.62}]},{"osd":2,"last update":"Tue Mar 26 02:22:33 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.60999999999999999,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.438,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.59899999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.41699999999999998}]},{"osd":4,"last update":"Tue Mar 26 02:22:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59299999999999997,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78400000000000003,"5min":0.878,"15min":0.878},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.58899999999999997,"15min":0.59299999999999997},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.80000000000000004,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.68600000000000005}]},{"osd":5,"last update":"Tue Mar 26 02:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.147,"5min":1.147,"15min":1.147},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.82599999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.78900000000000003}]},{"osd":6,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.63700000000000001}]},{"osd":7,"last update":"Tue Mar 26 02:23:07 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.65300000000000002}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5760,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739392,"statfs":{"total":95995035648,"available":95989137408,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.65100000000000002,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.53800000000000003,"15min":0.54200000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.72299999999999998,"5min":0.745,"15min":0.745},"last":0.70499999999999996}]},{"osd":1,"last update":"Tue Mar 26 02:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.41399999999999998,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.751,"5min":0.753,"15min":0.753},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.70699999999999996,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.61899999999999999}]},{"osd":2,"last update":"Tue Mar 26 02:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.55000000000000004,"15min":0.55100000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.74199999999999999}]},{"osd":3,"last update":"Tue Mar 26 02:22:55 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55600000000000005,"15min":0.56000000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.55100000000000005,"15min":0.55200000000000005},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.67700000000000005}]},{"osd":5,"last update":"Tue Mar 26 02:23:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.38900000000000001,"15min":0.375},"min":{"1min":0.29399999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.434},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.39900000000000002,"15min":0.38},"min":{"1min":0.30599999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.44700000000000001}]},{"osd":6,"last update":"Tue Mar 26 02:22:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.52900000000000003}]},{"osd":7,"last update":"Tue Mar 26 02:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.49099999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.42799999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.68500000000000005}]},{"osd":1,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56699999999999995,"15min":0.56599999999999995},"min":{"1min":0.41299999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.69399999999999995,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.53100000000000003,"15min":0.52700000000000002},"min":{"1min":0.39300000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.69799999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.71699999999999997}]},{"osd":2,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54000000000000004,"15min":0.53800000000000003},"min":{"1min":0.42799999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.44700000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.69599999999999995,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.78600000000000003}]},{"osd":3,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.47899999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.68500000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.45800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":0.69399999999999995,"5min":0.755,"15min":0.755},"last":0.76400000000000001}]},{"osd":4,"last update":"Tue Mar 26 02:23:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.42199999999999999,"15min":0.41499999999999998},"min":{"1min":0.33200000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.42899999999999999,"15min":0.42099999999999999},"min":{"1min":0.33400000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.59799999999999998}]},{"osd":6,"last update":"Tue Mar 26 02:22:39 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.64800000000000002}]},{"osd":7,"last update":"Tue Mar 26 02:23:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T02:23:32.057 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T02:23:32.057 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T02:23:32.057 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T02:23:32.058 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T02:23:32.274 INFO:teuthology.orchestra.run.smithi164.stderr:Inferring config /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:23:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:23:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:31] "GET /metrics HTTP/1.1" 200 191488 "" "Prometheus/2.33.4" 2024-03-26T02:23:32.676 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:32 smithi164 ceph-mon[24851]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:32.676 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:32 smithi164 ceph-mon[24851]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:23:32.677 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:32 smithi164 ceph-mon[29512]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:32.677 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:32 smithi164 ceph-mon[29512]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:23:33.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:32 smithi172 ceph-mon[25688]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:33.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:32 smithi172 ceph-mon[25688]: from='client.14601 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:23:34.181 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:23:34.181 INFO:teuthology.orchestra.run.smithi164.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T02:23:34.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:34 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2874691098' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:23:34.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:34 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2874691098' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:23:34.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:34 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2874691098' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:23:34.826 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T02:23:34.826 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T02:23:34.826 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T02:23:34.839 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi164.front.sepia.ceph.com 2024-03-26T02:23:34.839 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-03-26T02:23:35.481 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:35 smithi164 ceph-mon[24851]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:35.481 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:35 smithi164 ceph-mon[29512]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:35.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:35 smithi172 ceph-mon[25688]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:36.413 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:36 smithi164 ceph-mon[24851]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:36.414 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:36 smithi164 ceph-mon[29512]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:36.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:36 smithi172 ceph-mon[25688]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:38.296 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:23:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:23:38.322 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:23:38.322 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "ead3760d-b0da-4304-9dd2-a5729a384ff0", 2024-03-26T02:23:38.323 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "r", 2024-03-26T02:23:38.323 INFO:teuthology.orchestra.run.smithi164.stdout: "current_period": "4e5bd638-439b-4660-9806-f3d0696e8942", 2024-03-26T02:23:38.323 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1 2024-03-26T02:23:38.323 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:23:38.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:38 smithi172 ceph-mon[25688]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:38.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:38 smithi172 ceph-mon[25688]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:23:38.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:38 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2984919248' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:23:38.614 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:38 smithi164 ceph-mon[24851]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:38.614 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:38 smithi164 ceph-mon[24851]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:23:38.614 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:38 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2984919248' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:23:38.614 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:38 smithi164 ceph-mon[29512]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:38.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:38 smithi164 ceph-mon[29512]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:23:38.615 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:38 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2984919248' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:23:38.964 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-03-26T02:23:39.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:39 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2984919248' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:23:39.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:39 smithi164 ceph-mon[24851]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:23:39.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:39 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2984919248' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:23:39.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:39 smithi164 ceph-mon[29512]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:23:39.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:39 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2984919248' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:23:39.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:39 smithi172 ceph-mon[25688]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:23:40.357 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:40 smithi172 ceph-mon[25688]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:40.358 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:40 smithi172 ceph-mon[25688]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:23:40.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:40 smithi164 ceph-mon[24851]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:40.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:40 smithi164 ceph-mon[24851]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:23:40.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:40 smithi164 ceph-mon[29512]: pgmap v45: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:23:40.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:40 smithi164 ceph-mon[29512]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:23:40.709 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:23:40.709 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "2291a596-ccad-47c6-8027-6e6c72aa56ec", 2024-03-26T02:23:40.709 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "default", 2024-03-26T02:23:40.709 INFO:teuthology.orchestra.run.smithi164.stdout: "api_name": "default", 2024-03-26T02:23:40.709 INFO:teuthology.orchestra.run.smithi164.stdout: "is_master": "true", 2024-03-26T02:23:40.709 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [], 2024-03-26T02:23:40.709 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames": [], 2024-03-26T02:23:40.709 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames_s3website": [], 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "", 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout: "zones": [], 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_targets": [], 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout: "default_placement": "", 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "ead3760d-b0da-4304-9dd2-a5729a384ff0", 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_policy": { 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout: "groups": [] 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:40.710 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:23:41.331 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-03-26T02:23:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:23:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:41] "GET /metrics HTTP/1.1" 200 191491 "" "Prometheus/2.33.4" 2024-03-26T02:23:42.543 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:42 smithi172 ceph-mon[25688]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T02:23:42.595 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:42 smithi164 ceph-mon[29512]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T02:23:42.595 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:42 smithi164 ceph-mon[24851]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T02:23:42.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:42 smithi172 systemd[1]: Starting Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:23:43.106 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:23:43.106 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "2aa5e63b-e319-4790-8712-f178029fd6be", 2024-03-26T02:23:43.106 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "z", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "domain_root": "z.rgw.meta:root", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "control_pool": "z.rgw.control", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "gc_pool": "z.rgw.log:gc", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "lc_pool": "z.rgw.log:lc", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "log_pool": "z.rgw.log", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "roles_pool": "z.rgw.meta:roles", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "otp_pool": "z.rgw.otp", 2024-03-26T02:23:43.107 INFO:teuthology.orchestra.run.smithi164.stdout: "system_key": { 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "access_key": "", 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "secret_key": "" 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_pools": [ 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "default-placement", 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "val": { 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "index_pool": "z.rgw.buckets.index", 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": { 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD": { 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "data_pool": "z.rgw.buckets.data" 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:23:43.108 INFO:teuthology.orchestra.run.smithi164.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-03-26T02:23:43.109 INFO:teuthology.orchestra.run.smithi164.stdout: "index_type": 0 2024-03-26T02:23:43.109 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:43.109 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:43.109 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:23:43.109 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "ead3760d-b0da-4304-9dd2-a5729a384ff0", 2024-03-26T02:23:43.109 INFO:teuthology.orchestra.run.smithi164.stdout: "notif_pool": "z.rgw.log:notif" 2024-03-26T02:23:43.109 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:42 smithi172 podman[41872]: 2024-03-26 02:23:42.838636177 +0000 UTC m=+0.017436633 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:42 smithi172 podman[41872]: 2024-03-26 02:23:42.944142586 +0000 UTC m=+0.122943042 container create dfc5e042a9b59bab308b8a3b3b8d6042e3bf721640cbad2f063bcccb9245a927 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, version=8.5, architecture=x86_64, vendor=Red Hat, Inc., description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, io.openshift.expose-services=, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, distribution-scope=public, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 podman[41872]: 2024-03-26 02:23:43.155262275 +0000 UTC m=+0.334062734 container init dfc5e042a9b59bab308b8a3b3b8d6042e3bf721640cbad2f063bcccb9245a927 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, io.openshift.tags=base rhel8, version=8.5, com.redhat.component=ubi8-container, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, distribution-scope=public) 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 podman[41872]: 2024-03-26 02:23:43.157955807 +0000 UTC m=+0.336756265 container start dfc5e042a9b59bab308b8a3b3b8d6042e3bf721640cbad2f063bcccb9245a927 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, name=ubi8, io.openshift.tags=base rhel8, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T02:23:43.188 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T02:23:43.439 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T02:23:43.439 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T02:23:43.439 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-03-26T02:23:43.439 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-03-26T02:23:43.439 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-03-26T02:23:43.439 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-03-26T02:23:43.439 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="App mode production" logger=settings 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-03-26T02:23:43.440 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-03-26T02:23:43.441 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-03-26T02:23:43.442 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-03-26T02:23:43.442 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-03-26T02:23:43.442 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-03-26T02:23:43.443 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-03-26T02:23:43.444 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 bash[41872]: dfc5e042a9b59bab308b8a3b3b8d6042e3bf721640cbad2f063bcccb9245a927 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 systemd[1]: Started Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-03-26T02:23:43.445 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T02:23:43.446 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-03-26T02:23:43.447 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-03-26T02:23:43.448 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-03-26T02:23:43.449 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-03-26T02:23:43.450 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-03-26T02:23:43.451 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-03-26T02:23:43.452 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-03-26T02:23:43.452 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-03-26T02:23:43.689 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-03-26T02:23:43.689 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-03-26T02:23:43.689 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-03-26T02:23:43.689 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-03-26T02:23:43.690 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-03-26T02:23:43.691 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-03-26T02:23:43.694 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T02:23:43.694 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-03-26T02:23:43.694 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-03-26T02:23:43.694 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T02:23:43.695 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-03-26T02:23:43.695 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-03-26T02:23:43.695 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-03-26T02:23:43.695 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-03-26T02:23:43.695 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-03-26T02:23:43.695 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-03-26T02:23:43.695 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-03-26T02:23:43.695 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T02:23:43.696 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T02:23:43.696 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T02:23:43.697 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-03-26T02:23:43.698 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-03-26T02:23:43.699 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T02:23:43.700 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-03-26T02:23:43.701 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-03-26T02:23:43.702 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T02:23:43.703 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T02:23:43.704 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T02:23:43.705 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-03-26T02:23:43.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-03-26T02:23:43.707 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-03-26T02:23:43.708 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T02:23:43.709 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-03-26T02:23:43.709 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-03-26T02:23:43.709 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-03-26T02:23:43.709 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T02:23:43.709 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-03-26T02:23:43.709 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-03-26T02:23:43.709 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-03-26T02:23:43.709 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-03-26T02:23:43.852 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=522.512199ms 2024-03-26T02:23:43.954 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T02:23:43.955 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:23:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:23:43+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[24851]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[29512]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:44.534 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:44.535 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:44.535 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:44 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:44.688 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:44 smithi172 ceph-mon[25688]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-26T02:23:44.689 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:44 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:44.689 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:44 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:44.689 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:44 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:44.689 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:44 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:44.689 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:44 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:46.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:46 smithi172 ceph-mon[25688]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:23:46.724 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:46 smithi164 ceph-mon[24851]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:23:46.724 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:46 smithi164 ceph-mon[29512]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 897 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:23:47.529 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:47 smithi164 ceph-mon[29512]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:23:47.529 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:47 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2653024050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:23:47.529 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:47 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:23:47.529 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:47 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:47.530 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:47 smithi164 ceph-mon[24851]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:23:47.530 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:47 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2653024050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:23:47.530 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:47 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:23:47.530 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:47 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:47 smithi172 ceph-mon[25688]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:23:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:47 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2653024050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:23:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:47 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:23:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:47 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:48.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:23:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:23:48.410 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[24851]: pgmap v51: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[24851]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[24851]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[24851]: Reconfiguring daemon alertmanager.a on smithi164 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[29512]: pgmap v51: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[29512]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:23:48.411 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:48.412 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:48.412 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[29512]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:23:48.412 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:48 smithi164 ceph-mon[29512]: Reconfiguring daemon alertmanager.a on smithi164 2024-03-26T02:23:48.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:48 smithi172 ceph-mon[25688]: pgmap v51: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:23:48.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:48 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-03-26T02:23:48.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:48 smithi172 ceph-mon[25688]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:23:48.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:48 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:48.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:48 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:48.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:48 smithi172 ceph-mon[25688]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:23:48.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:48 smithi172 ceph-mon[25688]: Reconfiguring daemon alertmanager.a on smithi164 2024-03-26T02:23:49.665 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:49 smithi164 ceph-mon[24851]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:23:49.665 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:49 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2653024050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:23:49.665 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:49 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:23:49.666 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:49 smithi164 ceph-mon[29512]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:23:49.666 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:49 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2653024050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:23:49.666 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:49 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:23:49.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:49 smithi172 ceph-mon[25688]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:23:49.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:49 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2653024050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:23:49.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:49 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:23:50.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:50 smithi164 ceph-mon[24851]: pgmap v54: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T02:23:50.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:50 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-26T02:23:50.668 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:50 smithi164 ceph-mon[24851]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:23:50.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:50 smithi164 ceph-mon[29512]: pgmap v54: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T02:23:50.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:50 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-26T02:23:50.669 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:50 smithi164 ceph-mon[29512]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:23:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:50 smithi172 ceph-mon[25688]: pgmap v54: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-03-26T02:23:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:50 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-03-26T02:23:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:50 smithi172 ceph-mon[25688]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:23:51.045 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:50 smithi164 systemd[1]: Stopping Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:23:51.045 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:50 smithi164 bash[49333]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T02:23:51.294 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[47099]: level=info ts=2024-03-26T02:23:51.041Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:23:51.294 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 podman[49344]: 2024-03-26 02:23:51.062301464 +0000 UTC m=+0.139463946 container died c863bd7c0c497c4cb4a366e9421f11f3f1d8edfeb23d5a15bd882363807c0e5d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:51.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:51 smithi164 ceph-mon[24851]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:23:51.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:51 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:23:51.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:51 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:23:51.693 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:51 smithi164 ceph-mon[29512]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:23:51.694 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:51 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:23:51.694 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:51 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:23:51.694 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 podman[49344]: 2024-03-26 02:23:51.416941908 +0000 UTC m=+0.494104400 container cleanup c863bd7c0c497c4cb4a366e9421f11f3f1d8edfeb23d5a15bd882363807c0e5d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:51.694 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 bash[49344]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:23:51.694 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 bash[49370]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T02:23:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:51 smithi172 ceph-mon[25688]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:23:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:51 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:23:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:51 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:23:51.967 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 podman[49358]: 2024-03-26 02:23:51.692993195 +0000 UTC m=+0.626427945 container remove c863bd7c0c497c4cb4a366e9421f11f3f1d8edfeb23d5a15bd882363807c0e5d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:51.967 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 podman[49358]: 2024-03-26 02:23:51.760314099 +0000 UTC m=+0.693748849 volume remove b8736e9443305f0e40d52912da8d65031687b2f3b923dcc101ae8dc49745f0b3 2024-03-26T02:23:51.967 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:23:51.967 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 systemd[1]: Stopped Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:23:51.967 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 systemd[1]: Starting Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:23:52.283 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:23:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:23:51] "GET /metrics HTTP/1.1" 200 195279 "" "Prometheus/2.33.4" 2024-03-26T02:23:52.284 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 podman[49437]: 2024-03-26 02:23:51.888799206 +0000 UTC m=+0.018229259 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-26T02:23:52.284 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:51 smithi164 podman[49437]: 2024-03-26 02:23:51.996405153 +0000 UTC m=+0.125835189 volume create b619cd2ed5285507bb42c746c3b78f9bece4e83af8df2be8982e7a02b37dabad 2024-03-26T02:23:52.284 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 podman[49437]: 2024-03-26 02:23:52.094799164 +0000 UTC m=+0.224229199 container create 1fc3d79d0812a14cf4164c7b8c9071d6793b96fb8cb81efe5d02f999d3d5dfbc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[24851]: pgmap v57: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[24851]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[29512]: pgmap v57: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[29512]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:23:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 podman[49437]: 2024-03-26 02:23:52.405715339 +0000 UTC m=+0.535145377 container init 1fc3d79d0812a14cf4164c7b8c9071d6793b96fb8cb81efe5d02f999d3d5dfbc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 podman[49437]: 2024-03-26 02:23:52.408672067 +0000 UTC m=+0.538102116 container start 1fc3d79d0812a14cf4164c7b8c9071d6793b96fb8cb81efe5d02f999d3d5dfbc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:52.419Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:52.419Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:52.420Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.164 port=9094 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:52.421Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:52.446Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:52.446Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:52.448Z caller=main.go:518 msg=Listening address=:9093 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:52.448Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 bash[49437]: 1fc3d79d0812a14cf4164c7b8c9071d6793b96fb8cb81efe5d02f999d3d5dfbc 2024-03-26T02:23:52.796 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:52 smithi164 systemd[1]: Started Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:23:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:52 smithi172 ceph-mon[25688]: pgmap v57: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-26T02:23:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:52 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:23:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:52 smithi172 ceph-mon[25688]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:23:52.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:52 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:23:52.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:52 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "75eb23cd-c70f-4d08-950b-46a2f856e582", 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: "predecessor_uuid": "4e5bd638-439b-4660-9806-f3d0696e8942", 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_status": [], 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: "period_map": { 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "75eb23cd-c70f-4d08-950b-46a2f856e582", 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: "zonegroups": [ 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-03-26T02:23:53.467 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "2291a596-ccad-47c6-8027-6e6c72aa56ec", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "default", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "api_name": "default", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "is_master": "true", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [], 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames": [], 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "hostnames_s3website": [], 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "2aa5e63b-e319-4790-8712-f178029fd6be", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "zones": [ 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "id": "2aa5e63b-e319-4790-8712-f178029fd6be", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "z", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "endpoints": [], 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "log_meta": "false", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "log_data": "false", 2024-03-26T02:23:53.468 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_index_max_shards": 11, 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "read_only": "false", 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "tier_type": "", 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from_all": "true", 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_from": [], 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "redirect_zone": "" 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "placement_targets": [ 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "default-placement", 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "tags": [], 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "storage_classes": [ 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: "STANDARD" 2024-03-26T02:23:53.469 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: "default_placement": "default-placement", 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "ead3760d-b0da-4304-9dd2-a5729a384ff0", 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_policy": { 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: "groups": [] 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: "short_zone_ids": [ 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: "key": "2aa5e63b-e319-4790-8712-f178029fd6be", 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: "val": 971892034 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:53.470 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zonegroup": "2291a596-ccad-47c6-8027-6e6c72aa56ec", 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "master_zone": "2aa5e63b-e319-4790-8712-f178029fd6be", 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "period_config": { 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_quota": { 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "user_quota": { 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "check_on_raw": false, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size": -1, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "max_size_kb": 0, 2024-03-26T02:23:53.471 INFO:teuthology.orchestra.run.smithi164.stdout: "max_objects": -1 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "user_ratelimit": { 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "max_read_ops": 0, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "max_write_ops": 0, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "max_read_bytes": 0, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "max_write_bytes": 0, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "bucket_ratelimit": { 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "max_read_ops": 0, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "max_write_ops": 0, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "max_read_bytes": 0, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "max_write_bytes": 0, 2024-03-26T02:23:53.472 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "anonymous_ratelimit": { 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "max_read_ops": 0, 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "max_write_ops": 0, 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "max_read_bytes": 0, 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "max_write_bytes": 0, 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "enabled": false 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_id": "ead3760d-b0da-4304-9dd2-a5729a384ff0", 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_name": "r", 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout: "realm_epoch": 2 2024-03-26T02:23:53.473 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:23:53.504 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:53 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:23:53.504 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:53 smithi172 ceph-mon[25688]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:23:53.504 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:53 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T02:23:53.504 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:53 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T02:23:53.504 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:53 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:53.504 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:53 smithi172 ceph-mon[25688]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:23:53.504 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:53 smithi172 ceph-mon[25688]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:23:53.761 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[29512]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[29512]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[29512]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[24851]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/26772885' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-03-26T02:23:53.762 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:53.763 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[24851]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:23:53.763 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:53 smithi164 ceph-mon[24851]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:23:54.128 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-03-26T02:23:54.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:54 smithi172 ceph-mon[25688]: pgmap v60: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-26T02:23:54.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:54 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-26T02:23:54.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:54 smithi172 ceph-mon[25688]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:23:54.774 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:54 smithi164 ceph-mon[29512]: pgmap v60: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-26T02:23:54.774 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:54 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-26T02:23:54.774 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:54 smithi164 ceph-mon[29512]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:23:54.775 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:54 smithi164 ceph-mon[24851]: pgmap v60: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-03-26T02:23:54.775 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:54 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-03-26T02:23:54.775 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:54 smithi164 ceph-mon[24851]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:23:54.775 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:23:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:23:54.422Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00093156s 2024-03-26T02:23:55.879 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 systemd[1]: Stopping Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:23:55.880 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 bash[42496]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:23:56.123 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled rgw.foo update... 2024-03-26T02:23:56.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.878Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:23:56.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.879Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:23:56.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.879Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:23:56.195 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.879Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.879Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.879Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.879Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.879Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.879Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.880Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.880Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[40849]: ts=2024-03-26T02:23:55.880Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:55 smithi172 podman[42508]: 2024-03-26 02:23:55.889146131 +0000 UTC m=+0.112840856 container died 60410e1fd3de3250d161058eb833eeb94b708bc6a714127906ce90c6ba37d8b1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:56 smithi172 podman[42508]: 2024-03-26 02:23:56.195042856 +0000 UTC m=+0.418737590 container cleanup 60410e1fd3de3250d161058eb833eeb94b708bc6a714127906ce90c6ba37d8b1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:56.196 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:56 smithi172 bash[42508]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:23:56.446 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:56 smithi172 bash[42535]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:23:56.446 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:56 smithi172 podman[42523]: 2024-03-26 02:23:56.429581603 +0000 UTC m=+0.542170112 container remove 60410e1fd3de3250d161058eb833eeb94b708bc6a714127906ce90c6ba37d8b1 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:56.446 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:56 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T02:23:56.446 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:56 smithi172 systemd[1]: Stopped Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:23:56.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:56 smithi172 systemd[1]: Starting Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:23:56.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:56 smithi172 podman[42605]: 2024-03-26 02:23:56.56976259 +0000 UTC m=+0.018387859 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T02:23:56.810 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph osd pool create foo' 2024-03-26T02:23:57.368 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:57 smithi164 ceph-mon[29512]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-26T02:23:57.368 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:57 smithi164 ceph-mon[29512]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:23:57.369 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:57 smithi164 ceph-mon[29512]: Saving service rgw.foo spec with placement count:2 2024-03-26T02:23:57.369 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:57 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:57.369 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:57 smithi164 ceph-mon[24851]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-26T02:23:57.369 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:57 smithi164 ceph-mon[24851]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:23:57.369 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:57 smithi164 ceph-mon[24851]: Saving service rgw.foo spec with placement count:2 2024-03-26T02:23:57.369 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:57 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:57.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:57 smithi172 ceph-mon[25688]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-03-26T02:23:57.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:57 smithi172 ceph-mon[25688]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:23:57.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:57 smithi172 ceph-mon[25688]: Saving service rgw.foo spec with placement count:2 2024-03-26T02:23:57.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:57 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:23:58.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:57 smithi172 podman[42605]: 2024-03-26 02:23:57.660747977 +0000 UTC m=+1.109373240 container create cc5b561b62662704c658e9bad59cdfee4f4161087951003ae5eb31b8729d5df0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:59.283 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'foo' created 2024-03-26T02:23:59.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:59 smithi164 ceph-mon[24851]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:23:59.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:59 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1798037588' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T02:23:59.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:23:59 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T02:23:59.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:59 smithi164 ceph-mon[29512]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:23:59.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:59 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1798037588' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T02:23:59.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:23:59 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T02:23:59.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-mon[25688]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:23:59.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1798037588' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T02:23:59.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-03-26T02:23:59.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 podman[42605]: 2024-03-26 02:23:59.404351444 +0000 UTC m=+2.852976707 container init cc5b561b62662704c658e9bad59cdfee4f4161087951003ae5eb31b8729d5df0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:59.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 podman[42605]: 2024-03-26 02:23:59.408725427 +0000 UTC m=+2.857350690 container start cc5b561b62662704c658e9bad59cdfee4f4161087951003ae5eb31b8729d5df0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:23:59.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.435Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T02:23:59.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.435Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T02:23:59.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.435Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T02:23:59.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.435Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi172 (none))" 2024-03-26T02:23:59.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.435Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:23:59.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.435Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:23:59.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.437Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:23:59.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.437Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T02:23:59.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.438Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T02:23:59.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.441Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:23:59.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.441Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.901µs 2024-03-26T02:23:59.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.441Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:24:00.052 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'rbd pool init foo' 2024-03-26T02:24:00.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.643Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T02:24:00.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.643Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T02:24:00.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.643Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.789µs wal_replay_duration=202.209785ms total_replay_duration=202.250913ms 2024-03-26T02:24:00.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.645Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:24:00.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.645Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T02:24:00.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.645Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:24:00.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.660Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.557919ms db_storage=649ns remote_storage=1.194µs web_handler=326ns query_engine=720ns scrape=445.483µs scrape_sd=27.488µs notify=19.647µs notify_sd=5.79µs rules=14.821986ms 2024-03-26T02:24:00.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:23:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:23:59.660Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T02:24:00.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 909 B/s wr, 2 op/s 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.164:9093"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.164:9093"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.172:9095"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.172:9095"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:24:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.172:3000"}]: dispatch 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.172:3000"}]: dispatch 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 909 B/s wr, 2 op/s 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:24:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.164:9093"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.164:9093"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.172:9095"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.172:9095"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.172:3000"}]: dispatch 2024-03-26T02:24:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.172:3000"}]: dispatch 2024-03-26T02:24:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:00 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:00.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:24:00 smithi172 bash[42605]: cc5b561b62662704c658e9bad59cdfee4f4161087951003ae5eb31b8729d5df0 2024-03-26T02:24:00.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:24:00 smithi172 systemd[1]: Started Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:24:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 909 B/s wr, 2 op/s 2024-03-26T02:24:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-03-26T02:24:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:24:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:24:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:24:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.164:9093"}]: dispatch 2024-03-26T02:24:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.164:9093"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.172:9095"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.172:9095"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.172:3000"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.172:3000"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:00.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:00 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:01.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:01 smithi164 ceph-mon[24851]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:24:01.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:01 smithi164 ceph-mon[29512]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:24:01.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:01 smithi172 ceph-mon[25688]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:24:02.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:02 smithi164 ceph-mon[24851]: pgmap v67: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 911 B/s wr, 2 op/s 2024-03-26T02:24:02.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:02 smithi164 ceph-mon[24851]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:24:02.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:02 smithi164 ceph-mon[29512]: pgmap v67: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 911 B/s wr, 2 op/s 2024-03-26T02:24:02.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:02 smithi164 ceph-mon[29512]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:24:02.545 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:24:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:24:02.424Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002964216s 2024-03-26T02:24:02.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:02 smithi172 ceph-mon[25688]: pgmap v67: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 911 B/s rd, 911 B/s wr, 2 op/s 2024-03-26T02:24:02.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:02 smithi172 ceph-mon[25688]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:24:03.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:03 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1967511188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T02:24:03.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:03 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T02:24:03.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:03 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1967511188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T02:24:03.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:03 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T02:24:03.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:03 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1967511188' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T02:24:03.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:03 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-03-26T02:24:04.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:04 smithi172 ceph-mon[25688]: pgmap v69: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:24:04.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:04 smithi172 ceph-mon[25688]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-26T02:24:04.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:04 smithi172 ceph-mon[25688]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:24:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:04 smithi164 ceph-mon[24851]: pgmap v69: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:24:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:04 smithi164 ceph-mon[24851]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-26T02:24:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:04 smithi164 ceph-mon[24851]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:24:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:04 smithi164 ceph-mon[29512]: pgmap v69: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:24:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:04 smithi164 ceph-mon[29512]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-03-26T02:24:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:04 smithi164 ceph-mon[29512]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:24:05.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:05 smithi172 ceph-mon[25688]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:24:05.551 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:05 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:05.621 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:05 smithi164 ceph-mon[24851]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:24:05.621 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:05 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:05.622 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:05 smithi164 ceph-mon[29512]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:24:05.622 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:05 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch apply iscsi foo u p' 2024-03-26T02:24:06.574 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 205 B/s wr, 0 op/s 2024-03-26T02:24:06.574 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:24:06.574 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.574 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.574 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.574 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: Saving service rgw.foo spec with placement count:2 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[24851]: Deploying daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 205 B/s wr, 0 op/s 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.575 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: Saving service rgw.foo spec with placement count:2 2024-03-26T02:24:06.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:24:06.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:24:06.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:06.576 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:06 smithi164 ceph-mon[29512]: Deploying daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 205 B/s wr, 0 op/s 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: Saving service rgw.foo spec with placement count:2 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:06 smithi172 ceph-mon[25688]: Deploying daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:24:08.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:24:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:24:08.543 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:08 smithi164 ceph-mon[29512]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T02:24:08.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:08 smithi164 ceph-mon[24851]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T02:24:08.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:08 smithi172 ceph-mon[25688]: pgmap v74: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T02:24:09.421 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled iscsi.foo update... 2024-03-26T02:24:10.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T02:24:10.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:24:10.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: Saving service iscsi.foo spec with placement count:1 2024-03-26T02:24:10.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:24:10.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:24:10.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:10.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:10 smithi172 ceph-mon[25688]: Deploying daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: Saving service iscsi.foo spec with placement count:1 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.747 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[29512]: Deploying daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 170 B/s wr, 0 op/s 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: Saving service iscsi.foo spec with placement count:1 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:10.748 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:10 smithi164 ceph-mon[24851]: Deploying daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:24:10.769 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'sleep 180' 2024-03-26T02:24:12.154 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:24:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:11] "GET /metrics HTTP/1.1" 200 197898 "" "Prometheus/2.33.4" 2024-03-26T02:24:12.541 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:12 smithi172 ceph-mon[25688]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.4 KiB/s wr, 118 op/s 2024-03-26T02:24:12.674 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:12 smithi164 ceph-mon[24851]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.4 KiB/s wr, 118 op/s 2024-03-26T02:24:12.674 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:12 smithi164 ceph-mon[29512]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.4 KiB/s wr, 118 op/s 2024-03-26T02:24:13.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:13 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:13.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:13 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:13.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:13 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:13.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:13 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:13.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:13 smithi172 ceph-mon[25688]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:24:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[24851]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:13 smithi164 ceph-mon[29512]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:24:14.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:14 smithi172 ceph-mon[25688]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 117 op/s 2024-03-26T02:24:14.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:14 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:14.917 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:14 smithi164 ceph-mon[24851]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 117 op/s 2024-03-26T02:24:14.917 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:14 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:14.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:14 smithi164 ceph-mon[29512]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 4.2 KiB/s wr, 117 op/s 2024-03-26T02:24:14.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:14 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 301 op/s 2024-03-26T02:24:16.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: Checking dashboard <-> RGW credentials 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: Checking pool "foo" exists for service iscsi.foo 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 301 op/s 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.697 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: Checking dashboard <-> RGW credentials 2024-03-26T02:24:16.697 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.697 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.697 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.697 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: Checking pool "foo" exists for service iscsi.foo 2024-03-26T02:24:16.697 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:24:16.697 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:24:16.697 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:16 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 301 op/s 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: Checking dashboard <-> RGW credentials 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: Checking pool "foo" exists for service iscsi.foo 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:24:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:16 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:17.741 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:17 smithi164 ceph-mon[24851]: Deploying daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:24:17.741 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:17 smithi164 ceph-mon[29512]: Deploying daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:24:17.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:17 smithi172 ceph-mon[25688]: Deploying daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:24:18.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:24:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:24:18.727 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:18 smithi164 ceph-mon[24851]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 300 op/s 2024-03-26T02:24:18.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:18 smithi164 ceph-mon[29512]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 300 op/s 2024-03-26T02:24:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:18 smithi172 ceph-mon[25688]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 300 op/s 2024-03-26T02:24:19.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: mgrmap e20: y(active, since 119s), standbys: x 2024-03-26T02:24:19.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-03-26T02:24:19.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T02:24:19.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T02:24:19.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T02:24:19.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:19.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:19.587 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:19.587 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:19.587 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:24:19.587 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:24:19.587 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:19.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: mgrmap e20: y(active, since 119s), standbys: x 2024-03-26T02:24:19.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: mgrmap e20: y(active, since 119s), standbys: x 2024-03-26T02:24:19.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:24:19.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:20.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[24851]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 300 op/s 2024-03-26T02:24:20.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2024-03-26T02:24:20.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T02:24:20.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T02:24:20.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T02:24:20.779 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[24851]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3929222443' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2033405585' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/78250723"}]: dispatch 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[29512]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 300 op/s 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[29512]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3929222443' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:24:20.780 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:20 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2033405585' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/78250723"}]: dispatch 2024-03-26T02:24:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:20 smithi172 ceph-mon[25688]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 3.5 KiB/s wr, 300 op/s 2024-03-26T02:24:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2024-03-26T02:24:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T02:24:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T02:24:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:20 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T02:24:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:20 smithi172 ceph-mon[25688]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:24:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:20 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3929222443' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:24:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:20 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2033405585' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/78250723"}]: dispatch 2024-03-26T02:24:21.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:21 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2033405585' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/78250723"}]': finished 2024-03-26T02:24:21.691 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:21 smithi164 ceph-mon[24851]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:24:21.691 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:21 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/594169086' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3236129366"}]: dispatch 2024-03-26T02:24:21.691 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:21 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2033405585' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/78250723"}]': finished 2024-03-26T02:24:21.691 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:21 smithi164 ceph-mon[29512]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:24:21.691 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:21 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/594169086' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3236129366"}]: dispatch 2024-03-26T02:24:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:21 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2033405585' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/78250723"}]': finished 2024-03-26T02:24:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:21 smithi172 ceph-mon[25688]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:24:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:21 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/594169086' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3236129366"}]: dispatch 2024-03-26T02:24:22.278 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:24:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:21] "GET /metrics HTTP/1.1" 200 208071 "" "Prometheus/2.33.4" 2024-03-26T02:24:22.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[24851]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 511 B/s wr, 291 op/s 2024-03-26T02:24:22.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/594169086' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3236129366"}]': finished 2024-03-26T02:24:22.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[24851]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:24:22.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1081456701' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]: dispatch 2024-03-26T02:24:22.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]: dispatch 2024-03-26T02:24:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[29512]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 511 B/s wr, 291 op/s 2024-03-26T02:24:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/594169086' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3236129366"}]': finished 2024-03-26T02:24:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[29512]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:24:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1081456701' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]: dispatch 2024-03-26T02:24:22.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:22 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]: dispatch 2024-03-26T02:24:22.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:22 smithi172 ceph-mon[25688]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 178 KiB/s rd, 511 B/s wr, 291 op/s 2024-03-26T02:24:22.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:22 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/594169086' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3236129366"}]': finished 2024-03-26T02:24:22.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:22 smithi172 ceph-mon[25688]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:24:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:22 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1081456701' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]: dispatch 2024-03-26T02:24:22.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:22 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]: dispatch 2024-03-26T02:24:23.782 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]': finished 2024-03-26T02:24:23.782 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:23.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: Checking dashboard <-> RGW credentials 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2611092560' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/1272665921"}]: dispatch 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]': finished 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:24:23.784 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: Checking dashboard <-> RGW credentials 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2611092560' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/1272665921"}]: dispatch 2024-03-26T02:24:23.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2490859237"}]': finished 2024-03-26T02:24:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:24:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:24:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:24:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: Checking dashboard <-> RGW credentials 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2611092560' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/1272665921"}]: dispatch 2024-03-26T02:24:23.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:24:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[24851]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 767 B/s wr, 67 op/s 2024-03-26T02:24:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2611092560' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/1272665921"}]': finished 2024-03-26T02:24:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[24851]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:24:25.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1324387579' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]: dispatch 2024-03-26T02:24:25.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]: dispatch 2024-03-26T02:24:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[29512]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 767 B/s wr, 67 op/s 2024-03-26T02:24:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2611092560' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/1272665921"}]': finished 2024-03-26T02:24:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[29512]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:24:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1324387579' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]: dispatch 2024-03-26T02:24:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:24 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]: dispatch 2024-03-26T02:24:25.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:24 smithi172 ceph-mon[25688]: pgmap v87: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 47 KiB/s rd, 767 B/s wr, 67 op/s 2024-03-26T02:24:25.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:24 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2611092560' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/1272665921"}]': finished 2024-03-26T02:24:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:24 smithi172 ceph-mon[25688]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:24:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:24 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1324387579' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]: dispatch 2024-03-26T02:24:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:24 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]: dispatch 2024-03-26T02:24:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:25 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]': finished 2024-03-26T02:24:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:25 smithi164 ceph-mon[24851]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:24:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:25 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3169893386' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]: dispatch 2024-03-26T02:24:26.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:25 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]: dispatch 2024-03-26T02:24:26.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:25 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]': finished 2024-03-26T02:24:26.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:25 smithi164 ceph-mon[29512]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:24:26.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:25 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3169893386' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]: dispatch 2024-03-26T02:24:26.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:25 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]: dispatch 2024-03-26T02:24:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:25 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2200155095"}]': finished 2024-03-26T02:24:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:25 smithi172 ceph-mon[25688]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:24:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:25 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3169893386' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]: dispatch 2024-03-26T02:24:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:25 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]: dispatch 2024-03-26T02:24:27.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:26 smithi164 ceph-mon[29512]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 22 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T02:24:27.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:26 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]': finished 2024-03-26T02:24:27.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:26 smithi164 ceph-mon[29512]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T02:24:27.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:26 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2904249041' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3009346081"}]: dispatch 2024-03-26T02:24:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:26 smithi164 ceph-mon[24851]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 22 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T02:24:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:26 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]': finished 2024-03-26T02:24:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:26 smithi164 ceph-mon[24851]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T02:24:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:26 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2904249041' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3009346081"}]: dispatch 2024-03-26T02:24:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:26 smithi172 ceph-mon[25688]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 22 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T02:24:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:26 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1798451206"}]': finished 2024-03-26T02:24:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:26 smithi172 ceph-mon[25688]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T02:24:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:26 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2904249041' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3009346081"}]: dispatch 2024-03-26T02:24:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:27 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2904249041' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3009346081"}]': finished 2024-03-26T02:24:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:27 smithi164 ceph-mon[29512]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T02:24:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:27 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3409984138' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]: dispatch 2024-03-26T02:24:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:27 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]: dispatch 2024-03-26T02:24:28.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:27 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2904249041' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3009346081"}]': finished 2024-03-26T02:24:28.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:27 smithi164 ceph-mon[24851]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T02:24:28.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:27 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3409984138' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]: dispatch 2024-03-26T02:24:28.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:27 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]: dispatch 2024-03-26T02:24:28.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:24:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:24:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:27 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2904249041' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3009346081"}]': finished 2024-03-26T02:24:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:27 smithi172 ceph-mon[25688]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T02:24:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:27 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3409984138' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]: dispatch 2024-03-26T02:24:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:27 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]: dispatch 2024-03-26T02:24:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:28 smithi172 ceph-mon[25688]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 22 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T02:24:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:28 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]': finished 2024-03-26T02:24:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:28 smithi172 ceph-mon[25688]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T02:24:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:28 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3465512763' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/711619639"}]: dispatch 2024-03-26T02:24:29.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:28 smithi164 ceph-mon[24851]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 22 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T02:24:29.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:28 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]': finished 2024-03-26T02:24:29.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:28 smithi164 ceph-mon[24851]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T02:24:29.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:28 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3465512763' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/711619639"}]: dispatch 2024-03-26T02:24:29.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:28 smithi164 ceph-mon[29512]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 22 op/s; 2 B/s, 0 objects/s recovering 2024-03-26T02:24:29.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:28 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3498887725"}]': finished 2024-03-26T02:24:29.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:28 smithi164 ceph-mon[29512]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T02:24:29.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:28 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3465512763' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/711619639"}]: dispatch 2024-03-26T02:24:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:29 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3465512763' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/711619639"}]': finished 2024-03-26T02:24:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:29 smithi172 ceph-mon[25688]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T02:24:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:29 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3765455776' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1487529581"}]: dispatch 2024-03-26T02:24:30.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:29 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3465512763' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/711619639"}]': finished 2024-03-26T02:24:30.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:29 smithi164 ceph-mon[24851]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T02:24:30.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:29 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3765455776' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1487529581"}]: dispatch 2024-03-26T02:24:30.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:29 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3465512763' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/711619639"}]': finished 2024-03-26T02:24:30.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:29 smithi164 ceph-mon[29512]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T02:24:30.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:29 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3765455776' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1487529581"}]: dispatch 2024-03-26T02:24:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:30 smithi172 ceph-mon[25688]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:24:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:30 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:30 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3765455776' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1487529581"}]': finished 2024-03-26T02:24:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:30 smithi172 ceph-mon[25688]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T02:24:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:30 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1414144229' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3486476943"}]: dispatch 2024-03-26T02:24:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[24851]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:24:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3765455776' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1487529581"}]': finished 2024-03-26T02:24:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[24851]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T02:24:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1414144229' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3486476943"}]: dispatch 2024-03-26T02:24:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[29512]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:24:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3765455776' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1487529581"}]': finished 2024-03-26T02:24:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[29512]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T02:24:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:30 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1414144229' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3486476943"}]: dispatch 2024-03-26T02:24:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:31] "GET /metrics HTTP/1.1" 200 208071 "" "Prometheus/2.33.4" 2024-03-26T02:24:32.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1414144229' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3486476943"}]': finished 2024-03-26T02:24:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-mon[24851]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T02:24:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/672478051' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]: dispatch 2024-03-26T02:24:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]: dispatch 2024-03-26T02:24:32.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1414144229' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3486476943"}]': finished 2024-03-26T02:24:32.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-mon[29512]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T02:24:32.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/672478051' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]: dispatch 2024-03-26T02:24:32.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:31 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]: dispatch 2024-03-26T02:24:32.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:31 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1414144229' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3486476943"}]': finished 2024-03-26T02:24:32.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:31 smithi172 ceph-mon[25688]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T02:24:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:31 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/672478051' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]: dispatch 2024-03-26T02:24:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:31 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]: dispatch 2024-03-26T02:24:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:32 smithi164 ceph-mon[24851]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:32 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]': finished 2024-03-26T02:24:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:32 smithi164 ceph-mon[24851]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T02:24:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:32 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3457017798' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2862881695"}]: dispatch 2024-03-26T02:24:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:32 smithi164 ceph-mon[29512]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:32 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]': finished 2024-03-26T02:24:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:32 smithi164 ceph-mon[29512]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T02:24:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:32 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3457017798' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2862881695"}]: dispatch 2024-03-26T02:24:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:32 smithi172 ceph-mon[25688]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:32 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/1272665921"}]': finished 2024-03-26T02:24:33.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:32 smithi172 ceph-mon[25688]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T02:24:33.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:32 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3457017798' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2862881695"}]: dispatch 2024-03-26T02:24:34.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:33 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3457017798' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2862881695"}]': finished 2024-03-26T02:24:34.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:33 smithi164 ceph-mon[24851]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T02:24:34.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:33 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2014671415' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2862881695"}]: dispatch 2024-03-26T02:24:34.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:33 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3457017798' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2862881695"}]': finished 2024-03-26T02:24:34.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:33 smithi164 ceph-mon[29512]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T02:24:34.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:33 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2014671415' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2862881695"}]: dispatch 2024-03-26T02:24:34.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:33 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3457017798' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2862881695"}]': finished 2024-03-26T02:24:34.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:33 smithi172 ceph-mon[25688]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T02:24:34.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:33 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2014671415' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2862881695"}]: dispatch 2024-03-26T02:24:35.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[24851]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:35.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2014671415' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2862881695"}]': finished 2024-03-26T02:24:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[24851]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T02:24:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2457540402' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]: dispatch 2024-03-26T02:24:35.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]: dispatch 2024-03-26T02:24:35.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[29512]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:35.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2014671415' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2862881695"}]': finished 2024-03-26T02:24:35.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[29512]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T02:24:35.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2457540402' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]: dispatch 2024-03-26T02:24:35.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:34 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]: dispatch 2024-03-26T02:24:35.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:34 smithi172 ceph-mon[25688]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 75 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:35.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:34 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2014671415' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2862881695"}]': finished 2024-03-26T02:24:35.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:34 smithi172 ceph-mon[25688]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T02:24:35.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:34 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2457540402' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]: dispatch 2024-03-26T02:24:35.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:34 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]: dispatch 2024-03-26T02:24:36.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:35 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]': finished 2024-03-26T02:24:36.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:35 smithi164 ceph-mon[24851]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T02:24:36.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:35 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3693277847' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/75066052"}]: dispatch 2024-03-26T02:24:36.196 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:35 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]': finished 2024-03-26T02:24:36.196 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:35 smithi164 ceph-mon[29512]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T02:24:36.196 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:35 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3693277847' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/75066052"}]: dispatch 2024-03-26T02:24:36.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:35 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2200155095"}]': finished 2024-03-26T02:24:36.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:35 smithi172 ceph-mon[25688]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T02:24:36.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:35 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3693277847' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/75066052"}]: dispatch 2024-03-26T02:24:37.212 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:36 smithi164 ceph-mon[24851]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:37.212 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:36 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3693277847' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/75066052"}]': finished 2024-03-26T02:24:37.212 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:36 smithi164 ceph-mon[24851]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T02:24:37.212 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:36 smithi164 ceph-mon[29512]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:37.212 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:36 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3693277847' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/75066052"}]': finished 2024-03-26T02:24:37.213 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:36 smithi164 ceph-mon[29512]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T02:24:37.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:36 smithi172 ceph-mon[25688]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:37.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:36 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3693277847' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/75066052"}]': finished 2024-03-26T02:24:37.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:36 smithi172 ceph-mon[25688]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T02:24:38.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:24:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:24:39.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:38 smithi164 ceph-mon[24851]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:24:39.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:38 smithi164 ceph-mon[29512]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:24:39.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:38 smithi172 ceph-mon[25688]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:24:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:40 smithi164 ceph-mon[24851]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:40 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:41.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:40 smithi164 ceph-mon[29512]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:41.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:40 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:41.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:40 smithi172 ceph-mon[25688]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:41.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:40 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:24:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:41] "GET /metrics HTTP/1.1" 200 208165 "" "Prometheus/2.33.4" 2024-03-26T02:24:43.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:42 smithi164 ceph-mon[24851]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:24:43.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:42 smithi164 ceph-mon[29512]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:24:43.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:42 smithi172 ceph-mon[25688]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:24:44.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:44 smithi164 ceph-mon[24851]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:44.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:44 smithi164 ceph-mon[29512]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:44.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:44 smithi172 ceph-mon[25688]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:46 smithi164 ceph-mon[24851]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:24:46.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:46 smithi164 ceph-mon[29512]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:24:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:46 smithi172 ceph-mon[25688]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:24:48.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:24:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:24:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:48 smithi164 ceph-mon[24851]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-03-26T02:24:48.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:48 smithi164 ceph-mon[29512]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-03-26T02:24:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:48 smithi172 ceph-mon[25688]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 894 B/s rd, 0 op/s 2024-03-26T02:24:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:50 smithi164 ceph-mon[24851]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:50 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:50 smithi164 ceph-mon[29512]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:50 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:50 smithi172 ceph-mon[25688]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:50 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:24:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:51] "GET /metrics HTTP/1.1" 200 208123 "" "Prometheus/2.33.4" 2024-03-26T02:24:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:52 smithi164 ceph-mon[24851]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:52 smithi164 ceph-mon[29512]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:52 smithi172 ceph-mon[25688]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:54 smithi164 ceph-mon[24851]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:54.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:54 smithi164 ceph-mon[29512]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:54.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:54 smithi172 ceph-mon[25688]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:56 smithi164 ceph-mon[24851]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:56.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:56 smithi164 ceph-mon[29512]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:56 smithi172 ceph-mon[25688]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:58.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:24:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:24:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:24:58 smithi164 ceph-mon[24851]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:58.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:24:58 smithi164 ceph-mon[29512]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:24:58 smithi172 ceph-mon[25688]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:00 smithi164 ceph-mon[24851]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:00 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:00 smithi164 ceph-mon[29512]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:00 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:00 smithi172 ceph-mon[25688]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:00 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:25:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:01] "GET /metrics HTTP/1.1" 200 208123 "" "Prometheus/2.33.4" 2024-03-26T02:25:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:02 smithi164 ceph-mon[24851]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:02 smithi164 ceph-mon[29512]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:02 smithi172 ceph-mon[25688]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:04.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:04 smithi164 ceph-mon[24851]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:04.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:04 smithi164 ceph-mon[29512]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:04.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:04 smithi172 ceph-mon[25688]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:06.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:06 smithi164 ceph-mon[24851]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:06.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:06 smithi164 ceph-mon[29512]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:06 smithi172 ceph-mon[25688]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:08.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:25:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:25:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:08 smithi164 ceph-mon[24851]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:08.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:08 smithi164 ceph-mon[29512]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:08.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:08 smithi172 ceph-mon[25688]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:10 smithi164 ceph-mon[24851]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:10 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:10 smithi164 ceph-mon[29512]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:10 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:10.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:10 smithi172 ceph-mon[25688]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:10.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:10 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:25:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:11] "GET /metrics HTTP/1.1" 200 208087 "" "Prometheus/2.33.4" 2024-03-26T02:25:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:12 smithi164 ceph-mon[24851]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:12 smithi164 ceph-mon[29512]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:12.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:12 smithi172 ceph-mon[25688]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:14.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:14 smithi164 ceph-mon[24851]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:14.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:14 smithi164 ceph-mon[29512]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:14.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:14 smithi172 ceph-mon[25688]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:16.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:16 smithi164 ceph-mon[24851]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:16.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:16 smithi164 ceph-mon[29512]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:16 smithi172 ceph-mon[25688]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:18.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:25:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:25:18.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:18 smithi172 ceph-mon[25688]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:18.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:18 smithi164 ceph-mon[24851]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:18.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:18 smithi164 ceph-mon[29512]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:25:19.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:25:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:25:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:25:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:25:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:25:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:20 smithi164 ceph-mon[24851]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:20 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:20 smithi164 ceph-mon[29512]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:20 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:20 smithi172 ceph-mon[25688]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:20 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:25:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:21] "GET /metrics HTTP/1.1" 200 208064 "" "Prometheus/2.33.4" 2024-03-26T02:25:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:22 smithi164 ceph-mon[24851]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:22 smithi164 ceph-mon[29512]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:22 smithi172 ceph-mon[25688]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:23.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:25:23.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:25:23.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:25:23.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:23 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:25:23.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:25:23.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:25:23.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:25:23.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:23 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:25:23.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:25:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:25:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:25:23.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:23 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:25:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:24 smithi164 ceph-mon[24851]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:24.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:24 smithi164 ceph-mon[29512]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:24 smithi172 ceph-mon[25688]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:26.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:26 smithi164 ceph-mon[29512]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:26.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:26 smithi164 ceph-mon[24851]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:26 smithi172 ceph-mon[25688]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:28.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:25:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:25:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:28 smithi164 ceph-mon[24851]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:28 smithi164 ceph-mon[29512]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:28 smithi172 ceph-mon[25688]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:30 smithi164 ceph-mon[24851]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:30 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:30 smithi164 ceph-mon[29512]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:30 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:30 smithi172 ceph-mon[25688]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:30 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:25:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:31] "GET /metrics HTTP/1.1" 200 208064 "" "Prometheus/2.33.4" 2024-03-26T02:25:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:32 smithi164 ceph-mon[24851]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:32.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:32 smithi164 ceph-mon[29512]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:32 smithi172 ceph-mon[25688]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:34.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:34 smithi164 ceph-mon[24851]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:34.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:34 smithi164 ceph-mon[29512]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:34.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:34 smithi172 ceph-mon[25688]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:36 smithi164 ceph-mon[24851]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:36 smithi164 ceph-mon[29512]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:36 smithi172 ceph-mon[25688]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:38.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:25:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:25:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:38 smithi164 ceph-mon[24851]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:38.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:38 smithi164 ceph-mon[29512]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:38.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:38 smithi172 ceph-mon[25688]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:40 smithi164 ceph-mon[24851]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:40 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:40 smithi164 ceph-mon[29512]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:40 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:40 smithi172 ceph-mon[25688]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:40 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:25:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:41] "GET /metrics HTTP/1.1" 200 208075 "" "Prometheus/2.33.4" 2024-03-26T02:25:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:42 smithi164 ceph-mon[24851]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:42.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:42 smithi164 ceph-mon[29512]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:42.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:42 smithi172 ceph-mon[25688]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:44.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:44 smithi172 ceph-mon[25688]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:44 smithi164 ceph-mon[24851]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:44 smithi164 ceph-mon[29512]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:46 smithi164 ceph-mon[24851]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:46.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:46 smithi164 ceph-mon[29512]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:46 smithi172 ceph-mon[25688]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:48.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:25:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:25:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:48 smithi164 ceph-mon[24851]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:48.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:48 smithi164 ceph-mon[29512]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:48 smithi172 ceph-mon[25688]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:50 smithi164 ceph-mon[24851]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:50.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:50 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:50 smithi164 ceph-mon[29512]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:50 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:50 smithi172 ceph-mon[25688]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:50 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:25:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:51] "GET /metrics HTTP/1.1" 200 208070 "" "Prometheus/2.33.4" 2024-03-26T02:25:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:52 smithi164 ceph-mon[24851]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:52 smithi164 ceph-mon[29512]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:52 smithi172 ceph-mon[25688]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:54 smithi172 ceph-mon[25688]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:54 smithi164 ceph-mon[24851]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:54 smithi164 ceph-mon[29512]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:56 smithi164 ceph-mon[24851]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:56.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:56 smithi164 ceph-mon[29512]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:56 smithi172 ceph-mon[25688]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:58.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:25:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:25:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:25:58 smithi164 ceph-mon[24851]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:25:58 smithi164 ceph-mon[29512]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:25:58 smithi172 ceph-mon[25688]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:00 smithi164 ceph-mon[24851]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:00 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:00 smithi164 ceph-mon[29512]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:00 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:00 smithi172 ceph-mon[25688]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:00 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:26:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:01] "GET /metrics HTTP/1.1" 200 208070 "" "Prometheus/2.33.4" 2024-03-26T02:26:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:02 smithi164 ceph-mon[24851]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:02.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:02 smithi164 ceph-mon[29512]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:02 smithi172 ceph-mon[25688]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:04.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:04 smithi172 ceph-mon[25688]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:04 smithi164 ceph-mon[24851]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:04 smithi164 ceph-mon[29512]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:06.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:06 smithi164 ceph-mon[24851]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:06.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:06 smithi164 ceph-mon[29512]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:06 smithi172 ceph-mon[25688]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:08.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:26:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:26:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:08 smithi164 ceph-mon[24851]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:08.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:08 smithi164 ceph-mon[29512]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:08.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:08 smithi172 ceph-mon[25688]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:10 smithi164 ceph-mon[24851]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:10 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:10.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:10 smithi164 ceph-mon[29512]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:10 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:10.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:10 smithi172 ceph-mon[25688]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:10.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:10 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:26:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:11] "GET /metrics HTTP/1.1" 200 208078 "" "Prometheus/2.33.4" 2024-03-26T02:26:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:12 smithi164 ceph-mon[24851]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:12 smithi164 ceph-mon[29512]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:12.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:12 smithi172 ceph-mon[25688]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:14 smithi172 ceph-mon[25688]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:14.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:14 smithi164 ceph-mon[24851]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:14.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:14 smithi164 ceph-mon[29512]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:16.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:16 smithi164 ceph-mon[24851]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:16 smithi164 ceph-mon[29512]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:16 smithi172 ceph-mon[25688]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:18.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:26:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:26:18.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:18 smithi164 ceph-mon[24851]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:18.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:18 smithi164 ceph-mon[29512]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:18 smithi172 ceph-mon[25688]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:26:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:26:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:26:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:26:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:26:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:26:20.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:20 smithi164 ceph-mon[29512]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:20.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:20 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:20 smithi164 ceph-mon[24851]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:20.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:20 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:20 smithi172 ceph-mon[25688]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:20 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:26:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:21] "GET /metrics HTTP/1.1" 200 208074 "" "Prometheus/2.33.4" 2024-03-26T02:26:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:22 smithi164 ceph-mon[24851]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:22.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:22 smithi164 ceph-mon[29512]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:22 smithi172 ceph-mon[25688]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[24851]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[29512]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:26:24.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:26:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:26:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:26:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:24 smithi172 ceph-mon[25688]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:26:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:26 smithi164 ceph-mon[24851]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:26 smithi164 ceph-mon[29512]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:26.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:26 smithi172 ceph-mon[25688]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:28.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:26:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:26:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:28 smithi164 ceph-mon[24851]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:28 smithi164 ceph-mon[29512]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:28 smithi172 ceph-mon[25688]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:30 smithi164 ceph-mon[24851]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:30 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:30.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:30 smithi164 ceph-mon[29512]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:30 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:30 smithi172 ceph-mon[25688]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:30 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:26:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:31] "GET /metrics HTTP/1.1" 200 208074 "" "Prometheus/2.33.4" 2024-03-26T02:26:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:32 smithi164 ceph-mon[24851]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:32.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:32 smithi164 ceph-mon[29512]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:32 smithi172 ceph-mon[25688]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:34 smithi164 ceph-mon[24851]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:34 smithi164 ceph-mon[29512]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:34 smithi172 ceph-mon[25688]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:36 smithi164 ceph-mon[24851]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:36.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:36 smithi164 ceph-mon[29512]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:36 smithi172 ceph-mon[25688]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:38.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:26:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:26:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:38 smithi164 ceph-mon[24851]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:38.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:38 smithi164 ceph-mon[29512]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:38.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:38 smithi172 ceph-mon[25688]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:40 smithi164 ceph-mon[24851]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:40 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:40.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:40 smithi164 ceph-mon[29512]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:40 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:40 smithi172 ceph-mon[25688]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:40 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:26:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:41] "GET /metrics HTTP/1.1" 200 208067 "" "Prometheus/2.33.4" 2024-03-26T02:26:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:42 smithi164 ceph-mon[24851]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:42.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:42 smithi164 ceph-mon[29512]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:42.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:42 smithi172 ceph-mon[25688]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:44 smithi164 ceph-mon[24851]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:44.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:44 smithi164 ceph-mon[29512]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:44 smithi172 ceph-mon[25688]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:46 smithi164 ceph-mon[24851]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:46 smithi164 ceph-mon[29512]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:46 smithi172 ceph-mon[25688]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:48.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:26:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:26:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:48 smithi164 ceph-mon[24851]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:48.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:48 smithi164 ceph-mon[29512]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:48 smithi172 ceph-mon[25688]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:50 smithi164 ceph-mon[24851]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:50 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:50 smithi164 ceph-mon[29512]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:50 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:50 smithi172 ceph-mon[25688]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:50 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:26:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:51] "GET /metrics HTTP/1.1" 200 208020 "" "Prometheus/2.33.4" 2024-03-26T02:26:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:52 smithi164 ceph-mon[24851]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:52 smithi164 ceph-mon[29512]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:52 smithi172 ceph-mon[25688]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:54 smithi164 ceph-mon[24851]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:54.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:54 smithi164 ceph-mon[29512]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:54.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:54 smithi172 ceph-mon[25688]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:56 smithi164 ceph-mon[24851]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:56.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:56 smithi164 ceph-mon[29512]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:56 smithi172 ceph-mon[25688]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:58.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:26:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:26:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:26:58 smithi164 ceph-mon[24851]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:58.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:26:58 smithi164 ceph-mon[29512]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:26:58 smithi172 ceph-mon[25688]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:00 smithi164 ceph-mon[24851]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:00 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:00 smithi164 ceph-mon[29512]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:00 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:00 smithi172 ceph-mon[25688]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:00 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:27:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:01] "GET /metrics HTTP/1.1" 200 208020 "" "Prometheus/2.33.4" 2024-03-26T02:27:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:02 smithi164 ceph-mon[24851]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:02 smithi164 ceph-mon[29512]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:02 smithi172 ceph-mon[25688]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:04 smithi164 ceph-mon[24851]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:04 smithi164 ceph-mon[29512]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:04 smithi172 ceph-mon[25688]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:06.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:06 smithi164 ceph-mon[24851]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:06.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:06 smithi164 ceph-mon[29512]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:06 smithi172 ceph-mon[25688]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:08.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:27:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:27:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:08 smithi164 ceph-mon[24851]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:08.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:08 smithi164 ceph-mon[29512]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:08.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:08 smithi172 ceph-mon[25688]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:10 smithi164 ceph-mon[24851]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:10.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:10 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:10 smithi164 ceph-mon[29512]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:10 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:10.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:10 smithi172 ceph-mon[25688]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:10.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:10 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:27:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:11] "GET /metrics HTTP/1.1" 200 208021 "" "Prometheus/2.33.4" 2024-03-26T02:27:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:12 smithi164 ceph-mon[24851]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:12 smithi164 ceph-mon[29512]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:12.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:12 smithi172 ceph-mon[25688]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:13.330 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-26T02:27:14.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:14 smithi164 ceph-mon[24851]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:14.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:14 smithi164 ceph-mon[29512]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:14.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:14 smithi172 ceph-mon[25688]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:15.914 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-26T02:27:16.477 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:16 smithi164 ceph-mon[24851]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:16.477 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:16 smithi164 ceph-mon[29512]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:16.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:16 smithi172 ceph-mon[25688]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:18.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:27:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:27:18.369 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-26T02:27:18.672 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:18 smithi164 ceph-mon[24851]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:18.672 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:18 smithi164 ceph-mon[29512]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:18.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:18 smithi172 ceph-mon[25688]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:19.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:27:19.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:27:19.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:27:19.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:27:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:27:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:27:20.643 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:20 smithi164 ceph-mon[24851]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:20.643 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:20 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:20.643 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:20 smithi164 ceph-mon[29512]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:20.643 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:20 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:20 smithi172 ceph-mon[25688]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:20 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:20.993 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:27:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:27:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:21] "GET /metrics HTTP/1.1" 200 208021 "" "Prometheus/2.33.4" 2024-03-26T02:27:22.653 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:22 smithi164 ceph-mon[24851]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:22.653 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:22 smithi164 ceph-mon[29512]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:22 smithi172 ceph-mon[25688]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:22.849 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:27:22.849 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (3m) 3m ago 4m 15.4M - ba2b418f427c 1fc3d79d0812 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (3m) 3m ago 3m 47.3M - 8.3.5 dad864ee21e9 dfc5e042a9b5 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (3m) 3m ago 3m 73.7M - 3.5 e1d6a67b021e a39de1c9a6a4 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e 8879ebca8e08 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:9283 running (10m) 3m ago 10m 453M - 17.2.0 e1d6a67b021e bdfa3c50c167 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (10m) 3m ago 10m 47.3M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (8m) 3m ago 8m 39.6M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (8m) 3m ago 8m 38.9M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (4m) 3m ago 4m 18.1M - 1dbe0e931976 6fbe35cf8a99 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (4m) 3m ago 4m 15.4M - 1dbe0e931976 9c5963ac47a1 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (8m) 3m ago 8m 48.6M 4016M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (7m) 3m ago 7m 49.4M 4016M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:27:22.850 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (7m) 3m ago 7m 46.7M 4016M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:27:22.851 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (6m) 3m ago 6m 48.4M 4016M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:27:22.851 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (6m) 3m ago 6m 46.2M 4272M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:27:22.851 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (6m) 3m ago 6m 44.9M 4272M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:27:22.851 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (5m) 3m ago 5m 45.8M 4272M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:27:22.851 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (5m) 3m ago 5m 46.6M 4272M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:27:22.851 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (3m) 3m ago 4m 43.3M - 514e6a882f6e cc5b561b6266 2024-03-26T02:27:22.851 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (3m) 3m ago 3m 70.4M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:27:22.851 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (3m) 3m ago 3m 72.0M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:27:23.399 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions' 2024-03-26T02:27:23.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:23 smithi164 ceph-mon[24851]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:23.701 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:23 smithi164 ceph-mon[29512]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:23.705 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:23 smithi172 ceph-mon[25688]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:24.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:24 smithi164 ceph-mon[24851]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:24.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:24.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:27:24.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:24 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:27:24.690 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:24 smithi164 ceph-mon[29512]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:24.690 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:24.690 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:27:24.690 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:24 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:27:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:24 smithi172 ceph-mon[25688]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:27:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:24 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:27:25.397 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:27:25.398 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:27:25.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:27:25.399 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:27:25.399 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15 2024-03-26T02:27:25.399 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:27:25.399 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:27:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:25 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:27:25.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:25 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2162286557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:27:25.722 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:25 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:27:25.722 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:25 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2162286557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:27:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:25 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:27:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:25 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2162286557' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:27:26.028 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph -s' 2024-03-26T02:27:26.583 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:26 smithi164 ceph-mon[24851]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:26.583 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:26 smithi164 ceph-mon[29512]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:26.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:26 smithi172 ceph-mon[25688]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: cluster: 2024-03-26T02:27:27.969 INFO:teuthology.orchestra.run.smithi164.stdout: id: e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: health: HEALTH_OK 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: services: 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: mon: 3 daemons, quorum a,c,b (age 8m) 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: mgr: y(active, since 5m), standbys: x 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: osd: 8 osds: 8 up (since 5m), 8 in (since 5m) 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: data: 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: pools: 6 pools, 161 pgs 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: objects: 241 objects, 585 KiB 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: usage: 80 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:27:27.970 INFO:teuthology.orchestra.run.smithi164.stdout: pgs: 161 active+clean 2024-03-26T02:27:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:27:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: io: 2024-03-26T02:27:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-26T02:27:27.971 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:27:28.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:27:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:27:28.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:28 smithi164 ceph-mon[24851]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:28.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:28 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1295220216' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:27:28.694 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:28 smithi164 ceph-mon[29512]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:28.695 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:28 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1295220216' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:27:28.720 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ls' 2024-03-26T02:27:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:28 smithi172 ceph-mon[25688]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:28 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1295220216' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:27:30.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:30 smithi164 ceph-mon[24851]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:30.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:30 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:30 smithi164 ceph-mon[29512]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:30 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager ?:9093,9094 1/1 3m ago 5m smithi164=a;count:1 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:grafana ?:3000 1/1 3m ago 5m smithi172=a;count:1 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo 1/1 3m ago 3m count:1 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:mgr 2/2 3m ago 8m smithi164=y;smithi172=x;count:2 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:mon 3/3 3m ago 9m smithi164:172.21.15.164=a;smithi164:[v2:172.21.15.164:3301,v1:172.21.15.164:6790]=c;smithi172:172.21.15.172=b;count:3 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter ?:9100 2/2 3m ago 5m smithi164=a;smithi172=b;count:2 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:osd 8 3m ago - 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus ?:9095 1/1 3m ago 5m smithi172=a;count:1 2024-03-26T02:27:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo ?:8000 2/2 3m ago 3m count:2 2024-03-26T02:27:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:30 smithi172 ceph-mon[25688]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:30 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:31.259 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-26T02:27:31.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:31 smithi164 ceph-mon[24851]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:31.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:31 smithi164 ceph-mon[29512]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:31.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:31 smithi172 ceph-mon[25688]: from='client.24877 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:27:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:31] "GET /metrics HTTP/1.1" 200 208021 "" "Prometheus/2.33.4" 2024-03-26T02:27:32.793 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:32 smithi164 ceph-mon[29512]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:32 smithi164 ceph-mon[24851]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:32 smithi172 ceph-mon[25688]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:33.561 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled to redeploy mgr.x on host 'smithi172' 2024-03-26T02:27:33.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:33 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2480274963' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:27:33.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:33 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2480274963' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:27:33.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:33 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2480274963' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:27:34.196 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps --refresh' 2024-03-26T02:27:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[29512]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[29512]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:27:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[29512]: Schedule redeploy daemon mgr.x 2024-03-26T02:27:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[24851]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[24851]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:27:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[24851]: Schedule redeploy daemon mgr.x 2024-03-26T02:27:34.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:34 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:34 smithi172 ceph-mon[25688]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:34 smithi172 ceph-mon[25688]: from='client.24889 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.x", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:34 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:27:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:34 smithi172 ceph-mon[25688]: Schedule redeploy daemon mgr.x 2024-03-26T02:27:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:34 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:36.566 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (3m) 3m ago 4m 15.4M - ba2b418f427c 1fc3d79d0812 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (3m) 3m ago 3m 47.3M - 8.3.5 dad864ee21e9 dfc5e042a9b5 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (3m) 3m ago 3m 73.7M - 3.5 e1d6a67b021e a39de1c9a6a4 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e 8879ebca8e08 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:9283 running (10m) 3m ago 10m 453M - 17.2.0 e1d6a67b021e bdfa3c50c167 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (10m) 3m ago 10m 47.3M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (8m) 3m ago 8m 39.6M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (9m) 3m ago 9m 38.9M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (4m) 3m ago 4m 18.1M - 1dbe0e931976 6fbe35cf8a99 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (4m) 3m ago 4m 15.4M - 1dbe0e931976 9c5963ac47a1 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (8m) 3m ago 8m 48.6M 4016M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (7m) 3m ago 7m 49.4M 4016M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (7m) 3m ago 7m 46.7M 4016M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (7m) 3m ago 7m 48.4M 4016M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:27:36.567 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (6m) 3m ago 6m 46.2M 4272M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:27:36.568 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (6m) 3m ago 6m 44.9M 4272M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:27:36.568 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (6m) 3m ago 6m 45.8M 4272M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:27:36.568 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (5m) 3m ago 5m 46.6M 4272M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:27:36.568 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (3m) 3m ago 4m 43.3M - 514e6a882f6e cc5b561b6266 2024-03-26T02:27:36.568 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (3m) 3m ago 3m 70.4M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:27:36.568 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (3m) 3m ago 3m 72.0M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:27:36.583 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:36 smithi164 ceph-mon[24851]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:36.583 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:36 smithi164 ceph-mon[29512]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:36 smithi172 ceph-mon[25688]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:37.370 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'sleep 180' 2024-03-26T02:27:37.762 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:37 smithi164 ceph-mon[24851]: from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:37.763 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:37 smithi164 ceph-mon[29512]: from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:37.991 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:37 smithi172 ceph-mon[25688]: from='client.24895 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:27:38.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:27:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:27:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:38 smithi164 ceph-mon[24851]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:38 smithi164 ceph-mon[29512]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:38 smithi172 ceph-mon[25688]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:40.737 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:40 smithi164 ceph-mon[24851]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:40.737 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:40 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:40.738 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:40 smithi164 ceph-mon[29512]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:40.738 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:40 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:40 smithi172 ceph-mon[25688]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:40 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:27:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:41] "GET /metrics HTTP/1.1" 200 208008 "" "Prometheus/2.33.4" 2024-03-26T02:27:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:42 smithi164 ceph-mon[24851]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:42 smithi164 ceph-mon[29512]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:42 smithi172 ceph-mon[25688]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:44.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:44 smithi164 ceph-mon[29512]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:44.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:44 smithi164 ceph-mon[24851]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:44 smithi172 ceph-mon[25688]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:46 smithi164 ceph-mon[24851]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:46.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:46 smithi164 ceph-mon[29512]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:46 smithi172 ceph-mon[25688]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:48.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:27:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:27:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:48 smithi164 ceph-mon[24851]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:48.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:48 smithi164 ceph-mon[29512]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:48 smithi172 ceph-mon[25688]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:50 smithi164 ceph-mon[24851]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:50 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:50.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:50 smithi164 ceph-mon[29512]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:50 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:50 smithi172 ceph-mon[25688]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:50 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:27:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:51] "GET /metrics HTTP/1.1" 200 208014 "" "Prometheus/2.33.4" 2024-03-26T02:27:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:52 smithi164 ceph-mon[24851]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:52 smithi164 ceph-mon[29512]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:52.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:52 smithi172 ceph-mon[25688]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:54 smithi164 ceph-mon[24851]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:54.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:54 smithi164 ceph-mon[29512]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:54.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:54 smithi172 ceph-mon[25688]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:56 smithi164 ceph-mon[24851]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:56.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:56 smithi164 ceph-mon[29512]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:56 smithi172 ceph-mon[25688]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:58.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:27:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:27:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:27:58 smithi164 ceph-mon[24851]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:58.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:27:58 smithi164 ceph-mon[29512]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:27:58 smithi172 ceph-mon[25688]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:00.660 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:00 smithi164 ceph-mon[29512]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:00.661 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:00 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:00.661 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:00 smithi164 ceph-mon[24851]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:00.661 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:00 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:00 smithi172 ceph-mon[25688]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:00 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:28:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:01] "GET /metrics HTTP/1.1" 200 208014 "" "Prometheus/2.33.4" 2024-03-26T02:28:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[29512]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:02.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[24851]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:02.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:02 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:03.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:02 smithi172 ceph-mon[25688]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:03.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:02 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:03.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:02 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:03.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:02 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:03.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:02 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:03.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:03.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:03.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:03.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:28:04.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:04 smithi172 ceph-mon[25688]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:04.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:04 smithi164 ceph-mon[24851]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:04.683 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:04 smithi164 ceph-mon[29512]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[24851]: Deploying daemon mgr.x on smithi172 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.196 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:28:06.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:28:06.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:06.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:05 smithi164 ceph-mon[29512]: Deploying daemon mgr.x on smithi172 2024-03-26T02:28:06.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:05 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:05 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:05 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:05 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:28:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:05 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:28:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:05 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:05 smithi172 ceph-mon[25688]: Deploying daemon mgr.x on smithi172 2024-03-26T02:28:07.442 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:07 smithi164 ceph-mon[24851]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:07.443 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:07 smithi164 ceph-mon[29512]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:07.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:07 smithi172 ceph-mon[25688]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:08.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:28:09.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:09 smithi164 ceph-mon[24851]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:09.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:09 smithi164 ceph-mon[29512]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:09.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:09 smithi172 ceph-mon[25688]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:11.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:11 smithi164 ceph-mon[24851]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:11.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:11 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:11.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:11 smithi164 ceph-mon[29512]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:11.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:11 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:11.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:11 smithi172 ceph-mon[25688]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:11.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:11 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:28:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:11] "GET /metrics HTTP/1.1" 200 208019 "" "Prometheus/2.33.4" 2024-03-26T02:28:13.508 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:13 smithi164 ceph-mon[24851]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:13.508 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:13 smithi164 ceph-mon[29512]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:13.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:13 smithi172 ceph-mon[25688]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:13.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:28:13.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:13.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:28:13.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:14.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:14 smithi164 ceph-mon[24851]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:14.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:14 smithi164 ceph-mon[29512]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:14.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:14 smithi172 ceph-mon[25688]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:16.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:16 smithi164 ceph-mon[24851]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:16 smithi164 ceph-mon[29512]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:16 smithi172 ceph-mon[25688]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:18.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:28:18.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:18 smithi164 ceph-mon[24851]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:18.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:18 smithi164 ceph-mon[29512]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:18 smithi172 ceph-mon[25688]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:19.688 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:28:19.688 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:28:19.689 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:28:19.689 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:28:19.733 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:28:19.733 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:28:20.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:20 smithi164 ceph-mon[24851]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:20.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:20 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:20.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:20 smithi164 ceph-mon[29512]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:20.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:20 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:20 smithi172 ceph-mon[25688]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:20 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:28:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:21] "GET /metrics HTTP/1.1" 200 208037 "" "Prometheus/2.33.4" 2024-03-26T02:28:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:22 smithi164 ceph-mon[24851]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:22.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:22 smithi164 ceph-mon[29512]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:22 smithi172 ceph-mon[25688]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:23.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:28:23.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:23.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:23.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:28:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:24 smithi164 ceph-mon[24851]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:24 smithi164 ceph-mon[29512]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:24.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:24 smithi172 ceph-mon[25688]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:26 smithi164 ceph-mon[24851]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:26 smithi164 ceph-mon[29512]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:26 smithi172 ceph-mon[25688]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:28.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[26877]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:28:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:28 smithi164 ceph-mon[24851]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:28 smithi164 ceph-mon[29512]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:28 smithi172 ceph-mon[25688]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:30 smithi164 ceph-mon[24851]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:30 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:30 smithi164 ceph-mon[29512]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:30 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:30 smithi172 ceph-mon[25688]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:30 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:28:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:31] "GET /metrics HTTP/1.1" 200 208037 "" "Prometheus/2.33.4" 2024-03-26T02:28:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:32 smithi164 ceph-mon[24851]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:32.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:32 smithi164 ceph-mon[29512]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:32 smithi172 ceph-mon[25688]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:33.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:28:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:28:33.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:28:33.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:33.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:34 smithi164 ceph-mon[24851]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:34 smithi164 ceph-mon[29512]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:34 smithi172 ceph-mon[25688]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:36.510 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:36 smithi172 ceph-mon[25688]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:36.510 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:36 smithi172 systemd[1]: Stopping Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:28:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:36 smithi164 ceph-mon[24851]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:36 smithi164 ceph-mon[29512]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:36.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:36 smithi172 bash[45755]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr.x" found: no such container 2024-03-26T02:28:36.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:36 smithi172 podman[45765]: 2024-03-26 02:28:36.627258813 +0000 UTC m=+0.109654043 container died 8879ebca8e08e9292c522ff5b5c9b91284effd7ac4bee87460886cce61924a4d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_BRANCH=HEAD, distribution-scope=public, vcs-type=git, version=8, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, ceph=True, release=754, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True) 2024-03-26T02:28:37.191 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:36 smithi172 podman[45765]: 2024-03-26 02:28:36.956274525 +0000 UTC m=+0.438669734 container cleanup 8879ebca8e08e9292c522ff5b5c9b91284effd7ac4bee87460886cce61924a4d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, name=centos-stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, architecture=x86_64, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-type=git, io.openshift.expose-services=, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-26T02:28:37.191 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:36 smithi172 bash[45765]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x 2024-03-26T02:28:37.191 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:36 smithi172 bash[45791]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr.x" found: no such container 2024-03-26T02:28:37.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 podman[45779]: 2024-03-26 02:28:37.190894549 +0000 UTC m=+0.553938902 container remove 8879ebca8e08e9292c522ff5b5c9b91284effd7ac4bee87460886cce61924a4d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vendor=Red Hat, Inc., architecture=x86_64, name=centos-stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD) 2024-03-26T02:28:37.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:28:37.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-26T02:28:37.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 systemd[1]: Stopped Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:28:37.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service: Consumed 16.936s CPU time. 2024-03-26T02:28:37.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 systemd[1]: Starting Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:28:37.863 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 podman[45898]: 2024-03-26 02:28:37.511642545 +0000 UTC m=+0.017883686 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:28:37.863 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 podman[45898]: 2024-03-26 02:28:37.633806739 +0000 UTC m=+0.140047922 container create d86222d71593381cdcd0e0363f2dd66b5b4a5767cb06f3d3da98706a53622aba (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T02:28:37.863 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 podman[45898]: 2024-03-26 02:28:37.83616083 +0000 UTC m=+0.342401975 container init d86222d71593381cdcd0e0363f2dd66b5b4a5767cb06f3d3da98706a53622aba (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:28:37.864 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 podman[45898]: 2024-03-26 02:28:37.838687393 +0000 UTC m=+0.344928537 container start d86222d71593381cdcd0e0363f2dd66b5b4a5767cb06f3d3da98706a53622aba (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-26T02:28:38.006 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:37 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:37.740Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-26T02:28:38.259 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 ceph-mgr[45916]: -- 172.21.15.172:0/79691135 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5631d318e5a0 con 0x5631d3162800 2024-03-26T02:28:38.260 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 bash[45898]: d86222d71593381cdcd0e0363f2dd66b5b4a5767cb06f3d3da98706a53622aba 2024-03-26T02:28:38.260 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:37 smithi172 systemd[1]: Started Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:28:38.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:38.409+0000 7f9598eea200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:28:38.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:38.507+0000 7f9598eea200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:28:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[29512]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[24851]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:38 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-mon[25688]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:28:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:38 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:39.529 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:39.189+0000 7f9598eea200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:28:39.529 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:39.285+0000 7f9598eea200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:28:39.529 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:39.438+0000 7f9598eea200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:28:39.794 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:39.540+0000 7f9598eea200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:28:39.794 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:39.630+0000 7f9598eea200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:28:40.050 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:39.924+0000 7f9598eea200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:28:40.051 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:40.049+0000 7f9598eea200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:28:40.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:40.153+0000 7f9598eea200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:28:40.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:40.261+0000 7f9598eea200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:28:40.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:40.450+0000 7f9598eea200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:28:41.274 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:40 smithi172 ceph-mon[25688]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:41.274 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:40 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:40 smithi164 ceph-mon[24851]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:40 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:41.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:40 smithi164 ceph-mon[29512]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:41.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:40 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:41.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:41.463+0000 7f9598eea200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:28:41.871 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:41.598+0000 7f9598eea200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:28:42.252 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:41.955+0000 7f9598eea200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:28:42.252 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:42.049+0000 7f9598eea200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:28:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:28:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:41] "GET /metrics HTTP/1.1" 200 208039 "" "Prometheus/2.33.4" 2024-03-26T02:28:42.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:42.250+0000 7f9598eea200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:28:42.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:42.455+0000 7f9598eea200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:28:42.966 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:42.764+0000 7f9598eea200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:28:43.267 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:42 smithi172 ceph-mon[25688]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:43.267 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:42 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:43.267 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:42 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:43.268 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:43.025+0000 7f9598eea200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:28:43.268 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:43.266+0000 7f9598eea200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:28:43.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:42 smithi164 ceph-mon[24851]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:43.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:42 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:43.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:42 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:43.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:42 smithi164 ceph-mon[29512]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:43.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:42 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:43.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:42 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:28:43.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:28:43.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:28:43.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-26T02:28:43.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:28:44.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:44 smithi172 ceph-mon[25688]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:44.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:44.088+0000 7f9598eea200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:28:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:44 smithi164 ceph-mon[24851]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:44 smithi164 ceph-mon[29512]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:44.988 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:44.664+0000 7f9598eea200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:28:44.988 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:44.764+0000 7f9598eea200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:28:44.988 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:44.986+0000 7f9598eea200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:28:45.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:28:45] ENGINE Bus STARTING 2024-03-26T02:28:45.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: CherryPy Checker: 2024-03-26T02:28:45.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: The Application mounted at '' has an empty config. 2024-03-26T02:28:45.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:28:45.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:28:45] ENGINE Serving on http://:::9283 2024-03-26T02:28:45.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:28:45] ENGINE Bus STARTED 2024-03-26T02:28:45.782 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[24851]: Standby manager daemon x restarted 2024-03-26T02:28:45.782 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[24851]: Standby manager daemon x started 2024-03-26T02:28:45.782 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:28:45.783 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:28:45.783 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:28:45.783 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:28:45.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[29512]: Standby manager daemon x restarted 2024-03-26T02:28:45.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[29512]: Standby manager daemon x started 2024-03-26T02:28:45.784 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:28:45.784 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:28:45.784 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:28:45.784 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:28:45.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-mon[25688]: Standby manager daemon x restarted 2024-03-26T02:28:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-mon[25688]: Standby manager daemon x started 2024-03-26T02:28:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:28:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:28:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:28:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:45 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:28:46.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:45.781Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:46 smithi164 ceph-mon[24851]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-26T02:28:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:46 smithi164 ceph-mon[24851]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:46 smithi164 ceph-mon[29512]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-26T02:28:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:46 smithi164 ceph-mon[29512]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:46 smithi172 ceph-mon[25688]: mgrmap e21: y(active, since 6m), standbys: x 2024-03-26T02:28:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:46 smithi172 ceph-mon[25688]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:48.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:28:48.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:48 smithi164 ceph-mon[24851]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:48.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:48 smithi164 ceph-mon[29512]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:48.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:48 smithi172 ceph-mon[25688]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:50 smithi164 ceph-mon[24851]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:50 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:50 smithi164 ceph-mon[29512]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:50 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:50 smithi172 ceph-mon[25688]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:50 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:28:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:51] "GET /metrics HTTP/1.1" 200 208034 "" "Prometheus/2.33.4" 2024-03-26T02:28:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:52 smithi164 ceph-mon[24851]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:52 smithi164 ceph-mon[29512]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:52 smithi172 ceph-mon[25688]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:53.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:28:53.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:53.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:53.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:28:53.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:28:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:28:53.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:28:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:54 smithi164 ceph-mon[24851]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:54.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:54 smithi164 ceph-mon[29512]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:54.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:54 smithi172 ceph-mon[25688]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:56 smithi164 ceph-mon[24851]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:56.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:56 smithi164 ceph-mon[29512]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:56 smithi172 ceph-mon[25688]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:58.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:28:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:28:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:28:58 smithi164 ceph-mon[24851]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:58.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:28:58 smithi164 ceph-mon[29512]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:28:58 smithi172 ceph-mon[25688]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:00 smithi164 ceph-mon[24851]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:00 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:00 smithi164 ceph-mon[29512]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:00 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:00 smithi172 ceph-mon[25688]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:00 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:29:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:01] "GET /metrics HTTP/1.1" 200 208034 "" "Prometheus/2.33.4" 2024-03-26T02:29:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:02 smithi164 ceph-mon[24851]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:02 smithi164 ceph-mon[29512]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:02 smithi172 ceph-mon[25688]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:03.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:29:03.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:03.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:03.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:03.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:03.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:04 smithi164 ceph-mon[24851]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:04 smithi164 ceph-mon[29512]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:04 smithi172 ceph-mon[25688]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:06.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:06 smithi164 ceph-mon[24851]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:06.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:06 smithi164 ceph-mon[29512]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:06 smithi172 ceph-mon[25688]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:08.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:29:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:29:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:08 smithi164 ceph-mon[24851]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:08.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:08 smithi164 ceph-mon[29512]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:08.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:08 smithi172 ceph-mon[25688]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:10 smithi164 ceph-mon[24851]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:10 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:10 smithi164 ceph-mon[29512]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:10 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:10.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:10 smithi172 ceph-mon[25688]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:10.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:10 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:29:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:11] "GET /metrics HTTP/1.1" 200 208039 "" "Prometheus/2.33.4" 2024-03-26T02:29:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:12 smithi164 ceph-mon[24851]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:12.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:12 smithi164 ceph-mon[29512]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:12.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:12 smithi172 ceph-mon[25688]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:13.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:29:13.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:13.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:13.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:13.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:13.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:14.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:14 smithi164 ceph-mon[24851]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:14.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:14 smithi164 ceph-mon[29512]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:14.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:14 smithi172 ceph-mon[25688]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:16.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:16 smithi164 ceph-mon[24851]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:16 smithi164 ceph-mon[29512]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:16 smithi172 ceph-mon[25688]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:18.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:29:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:29:18.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:18 smithi164 ceph-mon[24851]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:18.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:18 smithi164 ceph-mon[29512]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:18 smithi172 ceph-mon[25688]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:19.743 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:29:19.743 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:29:19.743 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:29:19.744 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:29:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:29:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:29:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:20 smithi164 ceph-mon[24851]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:20 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:20 smithi164 ceph-mon[29512]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:20 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:20 smithi172 ceph-mon[25688]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:20 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:29:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:21] "GET /metrics HTTP/1.1" 200 208025 "" "Prometheus/2.33.4" 2024-03-26T02:29:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:22 smithi164 ceph-mon[24851]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:22 smithi164 ceph-mon[29512]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:22 smithi172 ceph-mon[25688]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:23.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:29:23.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:23.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:23.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:23.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:23.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:24 smithi164 ceph-mon[24851]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:24 smithi164 ceph-mon[29512]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:24.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:24 smithi172 ceph-mon[25688]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:26 smithi164 ceph-mon[24851]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:26.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:26 smithi164 ceph-mon[29512]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:26 smithi172 ceph-mon[25688]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:28.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:29:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:29:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:28 smithi164 ceph-mon[24851]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:28 smithi164 ceph-mon[29512]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:28 smithi172 ceph-mon[25688]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:30 smithi164 ceph-mon[24851]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:30 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:30 smithi164 ceph-mon[29512]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:30 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:30 smithi172 ceph-mon[25688]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:30 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:29:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:31] "GET /metrics HTTP/1.1" 200 208025 "" "Prometheus/2.33.4" 2024-03-26T02:29:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:32 smithi164 ceph-mon[24851]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:32.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:32 smithi164 ceph-mon[29512]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:32.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:32 smithi172 ceph-mon[25688]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:33.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:29:33.514Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:33.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:33.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:33.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:33.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:34 smithi164 ceph-mon[24851]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:34 smithi164 ceph-mon[29512]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:34 smithi172 ceph-mon[25688]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:36 smithi164 ceph-mon[24851]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:36 smithi164 ceph-mon[29512]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:36 smithi172 ceph-mon[25688]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:38.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:29:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:29:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:38 smithi164 ceph-mon[24851]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:38.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:38 smithi164 ceph-mon[29512]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:38.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:38 smithi172 ceph-mon[25688]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:40 smithi164 ceph-mon[24851]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:40 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:40 smithi164 ceph-mon[29512]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:40 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:40 smithi172 ceph-mon[25688]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:40 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:29:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:41] "GET /metrics HTTP/1.1" 200 208035 "" "Prometheus/2.33.4" 2024-03-26T02:29:42.772 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:42 smithi164 ceph-mon[24851]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:42.772 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:42 smithi164 ceph-mon[29512]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:42.774 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:42 smithi172 ceph-mon[25688]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:43.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:29:43.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:29:43.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:29:43.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:29:43.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:29:43.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:29:43.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:29:43.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:29:43.799 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:29:43.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:43.799 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:43.799 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:43 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:29:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:43 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:29:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:43 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:29:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:43 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:29:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:44 smithi164 ceph-mon[24851]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:44.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:44 smithi164 ceph-mon[29512]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:44 smithi172 ceph-mon[25688]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:46 smithi164 ceph-mon[24851]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:46 smithi164 ceph-mon[29512]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:46 smithi172 ceph-mon[25688]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:48.310 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:29:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:29:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:48 smithi164 ceph-mon[24851]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:48.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:48 smithi164 ceph-mon[29512]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:48 smithi172 ceph-mon[25688]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:50 smithi164 ceph-mon[24851]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:50.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:50 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:50 smithi164 ceph-mon[29512]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:50 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:50 smithi172 ceph-mon[25688]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:50 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:29:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:51] "GET /metrics HTTP/1.1" 200 208039 "" "Prometheus/2.33.4" 2024-03-26T02:29:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:52 smithi164 ceph-mon[24851]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:52 smithi164 ceph-mon[29512]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:52 smithi172 ceph-mon[25688]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:53.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:29:53.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:53.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:53.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:29:53.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:29:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:29:53.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:29:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:54 smithi164 ceph-mon[24851]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:54 smithi164 ceph-mon[29512]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:54.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:54 smithi172 ceph-mon[25688]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:56 smithi164 ceph-mon[24851]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:56.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:56 smithi164 ceph-mon[29512]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:56 smithi172 ceph-mon[25688]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:58.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:29:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:29:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:29:58 smithi164 ceph-mon[24851]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:29:58 smithi164 ceph-mon[29512]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:58.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:29:58 smithi172 ceph-mon[25688]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:00 smithi164 ceph-mon[24851]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:00 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:00 smithi164 ceph-mon[24851]: overall HEALTH_OK 2024-03-26T02:30:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:00 smithi164 ceph-mon[29512]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:00 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:00 smithi164 ceph-mon[29512]: overall HEALTH_OK 2024-03-26T02:30:00.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:00 smithi172 ceph-mon[25688]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:00.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:00 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:00.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:00 smithi172 ceph-mon[25688]: overall HEALTH_OK 2024-03-26T02:30:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:01] "GET /metrics HTTP/1.1" 200 208039 "" "Prometheus/2.33.4" 2024-03-26T02:30:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:02 smithi164 ceph-mon[24851]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:02 smithi164 ceph-mon[29512]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:02 smithi172 ceph-mon[25688]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:03.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:30:03.517Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:30:03.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:03.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:30:03.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:03.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:04 smithi164 ceph-mon[24851]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:04 smithi164 ceph-mon[29512]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:04 smithi172 ceph-mon[25688]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:06.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:06 smithi164 ceph-mon[24851]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:06.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:06 smithi164 ceph-mon[29512]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:06 smithi172 ceph-mon[25688]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:08.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:30:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:08 smithi164 ceph-mon[24851]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:08.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:08 smithi164 ceph-mon[29512]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:08.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:08 smithi172 ceph-mon[25688]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:10 smithi164 ceph-mon[24851]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:10 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:10 smithi164 ceph-mon[29512]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:10 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:10.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:10 smithi172 ceph-mon[25688]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:10.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:10 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:11] "GET /metrics HTTP/1.1" 200 208045 "" "Prometheus/2.33.4" 2024-03-26T02:30:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:12 smithi164 ceph-mon[24851]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:12 smithi164 ceph-mon[29512]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:12.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:12 smithi172 ceph-mon[25688]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:13.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:30:13.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:13.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:13.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:30:13.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:13.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:14.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:14 smithi164 ceph-mon[24851]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:14.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:14 smithi164 ceph-mon[29512]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:14.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:14 smithi172 ceph-mon[25688]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:16.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:16 smithi164 ceph-mon[24851]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:16 smithi164 ceph-mon[29512]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:16 smithi172 ceph-mon[25688]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:18.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:30:18.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:18 smithi164 ceph-mon[24851]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:18.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:18 smithi164 ceph-mon[29512]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:18 smithi172 ceph-mon[25688]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:19.789 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:19.790 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:19 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:19.790 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:19.790 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:19 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:19 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:20 smithi164 ceph-mon[24851]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:20.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:20 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:20 smithi164 ceph-mon[29512]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:20 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:20 smithi172 ceph-mon[25688]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:20 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:21] "GET /metrics HTTP/1.1" 200 208044 "" "Prometheus/2.33.4" 2024-03-26T02:30:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:22 smithi164 ceph-mon[24851]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:22.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:22 smithi164 ceph-mon[29512]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:22 smithi172 ceph-mon[25688]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:23.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:30:23.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:23.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:23.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:30:23.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:24 smithi164 ceph-mon[24851]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:24.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:24 smithi164 ceph-mon[29512]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:24.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:24 smithi172 ceph-mon[25688]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:26 smithi164 ceph-mon[24851]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:26 smithi164 ceph-mon[29512]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:26 smithi172 ceph-mon[25688]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:28.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:30:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:28 smithi164 ceph-mon[24851]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:28 smithi164 ceph-mon[29512]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:28 smithi172 ceph-mon[25688]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:30 smithi164 ceph-mon[24851]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:30 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:30 smithi164 ceph-mon[29512]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:30 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:30 smithi172 ceph-mon[25688]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:30 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:31] "GET /metrics HTTP/1.1" 200 208044 "" "Prometheus/2.33.4" 2024-03-26T02:30:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:32 smithi164 ceph-mon[24851]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:32.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:32 smithi164 ceph-mon[29512]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:32 smithi172 ceph-mon[25688]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:33.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:30:33.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:33.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:33.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:30:33.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:33.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:34 smithi164 ceph-mon[24851]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:34 smithi164 ceph-mon[29512]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:34 smithi172 ceph-mon[25688]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:36 smithi164 ceph-mon[24851]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:36.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:36 smithi164 ceph-mon[29512]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:36 smithi172 ceph-mon[25688]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:38.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:30:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:38 smithi164 ceph-mon[24851]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:38.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:38 smithi164 ceph-mon[29512]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:38.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:38 smithi172 ceph-mon[25688]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:40 smithi164 ceph-mon[24851]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:40 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:40 smithi164 ceph-mon[29512]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:40 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:40 smithi172 ceph-mon[25688]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:40 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:41.667 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:30:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:41] "GET /metrics HTTP/1.1" 200 208034 "" "Prometheus/2.33.4" 2024-03-26T02:30:42.678 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:42 smithi164 ceph-mon[24851]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:42.678 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:42 smithi164 ceph-mon[29512]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:42.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:42 smithi172 ceph-mon[25688]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:43.783 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:30:43.783 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (6m) 2m ago 7m 24.7M - ba2b418f427c 1fc3d79d0812 2024-03-26T02:30:43.783 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (7m) 2m ago 7m 56.7M - 8.3.5 dad864ee21e9 dfc5e042a9b5 2024-03-26T02:30:43.783 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (6m) 2m ago 6m 55.0M - 3.5 e1d6a67b021e a39de1c9a6a4 2024-03-26T02:30:43.783 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283 running (2m) 2m ago 11m 66.4M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 d86222d71593 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:9283 running (13m) 2m ago 13m 459M - 17.2.0 e1d6a67b021e bdfa3c50c167 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (13m) 2m ago 13m 50.2M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (12m) 2m ago 12m 41.0M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (12m) 2m ago 12m 36.3M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (8m) 2m ago 8m 19.4M - 1dbe0e931976 6fbe35cf8a99 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (7m) 2m ago 7m 16.7M - 1dbe0e931976 9c5963ac47a1 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (11m) 2m ago 11m 50.0M 4016M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (11m) 2m ago 11m 50.9M 4016M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (10m) 2m ago 10m 49.3M 4016M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (10m) 2m ago 10m 51.6M 4016M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (9m) 2m ago 9m 49.9M 4272M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (9m) 2m ago 9m 48.6M 4272M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:30:43.784 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (9m) 2m ago 9m 49.1M 4272M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:30:43.785 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (8m) 2m ago 8m 49.0M 4272M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:30:43.785 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (6m) 2m ago 8m 66.4M - 514e6a882f6e cc5b561b6266 2024-03-26T02:30:43.785 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (6m) 2m ago 6m 71.5M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:30:43.785 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (6m) 2m ago 6m 73.5M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:30:43.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:43.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:30:43.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:30:43.785 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-mon[24851]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:30:43.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:43.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:30:43.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:30:43.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-mon[29512]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:30:43.786 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:30:43.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:30:43.786 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:43.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:30:43.786 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:43.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:43 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:43 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:30:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:43 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:30:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:43 smithi172 ceph-mon[25688]: from='mgr.14421 172.21.15.164:0/2797718355' entity='mgr.y' 2024-03-26T02:30:44.425 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions' 2024-03-26T02:30:44.737 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:44 smithi164 ceph-mon[24851]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:44.738 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:44 smithi164 ceph-mon[24851]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:30:44.738 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:44 smithi164 ceph-mon[29512]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:44.738 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:44 smithi164 ceph-mon[29512]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:30:44.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:44 smithi172 ceph-mon[25688]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:44.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:44 smithi172 ceph-mon[25688]: from='client.15003 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 1 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:30:46.534 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 1 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:30:46.535 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:30:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:46 smithi164 ceph-mon[24851]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:46 smithi164 ceph-mon[29512]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:46 smithi172 ceph-mon[25688]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:47.133 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph -s' 2024-03-26T02:30:47.760 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:47 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2679752806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:30:47.761 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:47 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2679752806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:30:47.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:47 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2679752806' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:30:48.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:30:48.582 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:48 smithi164 ceph-mon[24851]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:48.582 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:48 smithi164 ceph-mon[29512]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:48 smithi172 ceph-mon[25688]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: cluster: 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: id: e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: health: HEALTH_OK 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: services: 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: mon: 3 daemons, quorum a,c,b (age 12m) 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: mgr: y(active, since 8m), standbys: x 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: osd: 8 osds: 8 up (since 8m), 8 in (since 9m) 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:30:49.037 INFO:teuthology.orchestra.run.smithi164.stdout: data: 2024-03-26T02:30:49.038 INFO:teuthology.orchestra.run.smithi164.stdout: pools: 6 pools, 161 pgs 2024-03-26T02:30:49.038 INFO:teuthology.orchestra.run.smithi164.stdout: objects: 241 objects, 585 KiB 2024-03-26T02:30:49.038 INFO:teuthology.orchestra.run.smithi164.stdout: usage: 80 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:30:49.038 INFO:teuthology.orchestra.run.smithi164.stdout: pgs: 161 active+clean 2024-03-26T02:30:49.038 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:30:49.038 INFO:teuthology.orchestra.run.smithi164.stdout: io: 2024-03-26T02:30:49.038 INFO:teuthology.orchestra.run.smithi164.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-26T02:30:49.038 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:30:49.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:49 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3910636876' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:30:49.724 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:49 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3910636876' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:30:49.745 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:30:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:49 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3910636876' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:30:50.752 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:50 smithi164 ceph-mon[24851]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:50.753 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:50 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:50.753 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:50 smithi164 ceph-mon[29512]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:50.753 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:50 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:50 smithi172 ceph-mon[25688]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:50 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:51.781 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:30:52.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:51] "GET /metrics HTTP/1.1" 200 208032 "" "Prometheus/2.33.4" 2024-03-26T02:30:52.469 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 2'"'"'' 2024-03-26T02:30:52.774 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:52 smithi164 ceph-mon[24851]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:52.774 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:52 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3602070354' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:30:52.774 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:52 smithi164 ceph-mon[29512]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:52.774 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:52 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3602070354' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:30:52.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:52 smithi172 ceph-mon[25688]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:52.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:52 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3602070354' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:30:53.775 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:30:53.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:53.775 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:53.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:30:53.775 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:53.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:30:54.518 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:30:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:54 smithi164 ceph-mon[24851]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:54.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:54 smithi164 ceph-mon[29512]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:54.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:54 smithi172 ceph-mon[25688]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:55.132 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph mgr fail' 2024-03-26T02:30:55.678 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:55 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/4202006158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:30:55.679 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:55 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/4202006158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:30:55.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:55 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/4202006158' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:30:56.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:56 smithi164 ceph-mon[29512]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:56.723 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:56 smithi164 ceph-mon[24851]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:56 smithi172 ceph-mon[25688]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 80 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:57.629 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2929091145' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:30:57.629 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-mon[24851]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T02:30:57.630 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:57.381+0000 7f6c1674c700 -1 mgr handle_mgr_map I was active but no longer amignoring --setuser ceph since I am not root 2024-03-26T02:30:57.630 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ignoring --setgroup ceph since I am not root 2024-03-26T02:30:57.630 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-mgr[25091]: -- 172.21.15.164:0/518828300 <== mon.0 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55758d8b2340 con 0x55758d88ac00 2024-03-26T02:30:57.630 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2929091145' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:30:57.630 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-mon[29512]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T02:30:57.636 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2929091145' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:30:57.636 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-mon[25688]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T02:30:57.637 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:30:57] ENGINE Bus STOPPING 2024-03-26T02:30:57.637 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:30:57] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:30:57.637 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:30:57] ENGINE Bus STOPPED 2024-03-26T02:30:57.637 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:30:57] ENGINE Bus STARTING 2024-03-26T02:30:57.637 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:30:57] ENGINE Serving on http://:::9283 2024-03-26T02:30:57.637 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:30:57] ENGINE Bus STARTED 2024-03-26T02:30:58.048 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'sleep 180' 2024-03-26T02:30:58.252 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:57.900+0000 7f0e2f487000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:30:58.252 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:57 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:57.993+0000 7f0e2f487000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:30:58.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:30:57 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:30:57] "GET /metrics HTTP/1.1" 200 34904 "" "Prometheus/2.33.4" 2024-03-26T02:30:58.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:30:58.251Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2929091145' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: mgrmap e22: x(active, starting, since 0.23696s) 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:30:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: Manager daemon x is now available 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: Queued rgw.foo for migration 2024-03-26T02:30:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2929091145' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: mgrmap e22: x(active, starting, since 0.23696s) 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:30:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: Manager daemon x is now available 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: Queued rgw.foo for migration 2024-03-26T02:30:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[24851]: Deploying cephadm binary to smithi172 2024-03-26T02:30:58.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:58.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-mon[29512]: Deploying cephadm binary to smithi172 2024-03-26T02:30:58.658 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2929091145' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: mgrmap e22: x(active, starting, since 0.23696s) 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:30:58.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: Manager daemon x is now available 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: Queued rgw.foo for migration 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.660 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:30:58.661 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-mon[25688]: Deploying cephadm binary to smithi172 2024-03-26T02:30:58.945 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:58.619+0000 7f0e2f487000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:30:58.945 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:58.711+0000 7f0e2f487000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:30:58.945 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:58.794+0000 7f0e2f487000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:30:59.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:30:58.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: |\n OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up at {{ $value | humanize }} times once a minute for 5 minutes. This could indicate a network issue (latency, packet drop, disruption) on the clusters \"cluster network\". Check the network environment on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSD's to flap (mark each other out)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.164:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:30:59.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:30:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:30:58.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephPGImbalance\nexpr: abs(((ceph_osd_numpg > 0) - on(job) group_left() avg by(job) (ceph_osd_numpg\n > 0)) / on(job) group_left() avg by(job) (ceph_osd_numpg > 0)) * on(ceph_daemon)\n group_left(hostname) ceph_osd_metadata > 0.3\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.5\n severity: warning\n type: ceph_default\nannotations:\n description: |\n OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count.\n summary: PG allocations are not balanced across devices\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.164:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:30:59.222 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:58.943+0000 7f0e2f487000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:30:59.222 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:59.026+0000 7f0e2f487000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:30:59.222 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:59.136+0000 7f0e2f487000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:30:59.482 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:59.220+0000 7f0e2f487000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:30:59.483 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:59.389+0000 7f0e2f487000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:30:59.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[24851]: Deploying cephadm binary to smithi164 2024-03-26T02:30:59.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[24851]: [26/Mar/2024:02:30:58] ENGINE Bus STARTING 2024-03-26T02:30:59.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[24851]: [26/Mar/2024:02:30:58] ENGINE Serving on http://172.21.15.172:8765 2024-03-26T02:30:59.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[24851]: [26/Mar/2024:02:30:58] ENGINE Serving on https://172.21.15.172:7150 2024-03-26T02:30:59.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[24851]: [26/Mar/2024:02:30:58] ENGINE Bus STARTED 2024-03-26T02:30:59.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[24851]: mgrmap e23: x(active, since 1.24535s) 2024-03-26T02:30:59.483 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[24851]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:30:59.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[29512]: Deploying cephadm binary to smithi164 2024-03-26T02:30:59.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[29512]: [26/Mar/2024:02:30:58] ENGINE Bus STARTING 2024-03-26T02:30:59.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[29512]: [26/Mar/2024:02:30:58] ENGINE Serving on http://172.21.15.172:8765 2024-03-26T02:30:59.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[29512]: [26/Mar/2024:02:30:58] ENGINE Serving on https://172.21.15.172:7150 2024-03-26T02:30:59.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[29512]: [26/Mar/2024:02:30:58] ENGINE Bus STARTED 2024-03-26T02:30:59.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[29512]: mgrmap e23: x(active, since 1.24535s) 2024-03-26T02:30:59.484 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-mon[29512]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:30:59.741 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:30:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:30:59.480+0000 7f0e2f487000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:30:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:59 smithi172 ceph-mon[25688]: Deploying cephadm binary to smithi164 2024-03-26T02:30:59.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:59 smithi172 ceph-mon[25688]: [26/Mar/2024:02:30:58] ENGINE Bus STARTING 2024-03-26T02:30:59.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:59 smithi172 ceph-mon[25688]: [26/Mar/2024:02:30:58] ENGINE Serving on http://172.21.15.172:8765 2024-03-26T02:30:59.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:59 smithi172 ceph-mon[25688]: [26/Mar/2024:02:30:58] ENGINE Serving on https://172.21.15.172:7150 2024-03-26T02:30:59.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:59 smithi172 ceph-mon[25688]: [26/Mar/2024:02:30:58] ENGINE Bus STARTED 2024-03-26T02:30:59.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:59 smithi172 ceph-mon[25688]: mgrmap e23: x(active, since 1.24535s) 2024-03-26T02:30:59.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:30:59 smithi172 ceph-mon[25688]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:00.656 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:00 smithi164 ceph-mon[29512]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:00.657 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:00 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:00.657 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:00 smithi164 ceph-mon[24851]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:00.657 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:00 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:00.657 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:00 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:00.524+0000 7f0e2f487000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:31:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:00 smithi172 ceph-mon[25688]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:00 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:00.937 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:00 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:00.655+0000 7f0e2f487000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:31:00.937 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:00 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:00.848+0000 7f0e2f487000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:31:01.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:00 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:00.935+0000 7f0e2f487000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:31:01.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:01.074+0000 7f0e2f487000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:31:01.723 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:01 smithi164 ceph-mon[24851]: mgrmap e24: x(active, since 3s) 2024-03-26T02:31:01.723 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:01 smithi164 ceph-mon[29512]: mgrmap e24: x(active, since 3s) 2024-03-26T02:31:01.723 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:01.407+0000 7f0e2f487000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:31:01.724 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:01.635+0000 7f0e2f487000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:31:01.724 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:01.722+0000 7f0e2f487000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:31:01.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:01 smithi172 ceph-mon[25688]: mgrmap e24: x(active, since 3s) 2024-03-26T02:31:02.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:01.894+0000 7f0e2f487000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:31:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:02 smithi164 ceph-mon[24851]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:02 smithi164 ceph-mon[29512]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:02 smithi172 ceph-mon[25688]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:03.158 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:02.837+0000 7f0e2f487000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:31:03.158 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:03.157+0000 7f0e2f487000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:31:03.420 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:03.260+0000 7f0e2f487000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:31:03.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:31:03.476Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:31:03.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:31:03.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:31:03.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:31:03.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:31:03.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:31:03.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:31:03.795 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:03.419+0000 7f0e2f487000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:31:03.795 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:31:03] ENGINE Bus STARTING 2024-03-26T02:31:03.795 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: CherryPy Checker: 2024-03-26T02:31:03.795 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: The Application mounted at '' has an empty config. 2024-03-26T02:31:03.795 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: 2024-03-26T02:31:03.795 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:31:03] ENGINE Serving on http://:::9283 2024-03-26T02:31:03.795 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:03 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: [26/Mar/2024:02:31:03] ENGINE Bus STARTED 2024-03-26T02:31:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[29512]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[29512]: Standby manager daemon y started 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[24851]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[24851]: Standby manager daemon y started 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:31:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:04 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:31:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:04 smithi172 ceph-mon[25688]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:31:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:04 smithi172 ceph-mon[25688]: Standby manager daemon y started 2024-03-26T02:31:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:04 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:31:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:04 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:31:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:04 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:31:04.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:04 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/2163320802' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:31:05.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:05 smithi164 ceph-mon[24851]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-26T02:31:05.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:05 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:31:05.704 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:05 smithi164 ceph-mon[29512]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-26T02:31:05.705 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:05 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:31:05.732 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:05 smithi172 ceph-mon[25688]: mgrmap e25: x(active, since 7s), standbys: y 2024-03-26T02:31:05.732 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:05 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:31:06.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:06 smithi164 ceph-mon[29512]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:31:06.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:06 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:06.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:06 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:06.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:06 smithi164 ceph-mon[24851]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:31:06.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:06 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:06.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:06 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:06 smithi172 ceph-mon[25688]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:31:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:06 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:06.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:06 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:07 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:31:07] "GET /metrics HTTP/1.1" 200 34904 "" "Prometheus/2.33.4" 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:08.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:07 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi164 to 2992M 2024-03-26T02:31:09.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: Adjusting osd_memory_target on smithi172 to 3248M 2024-03-26T02:31:09.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi164 to 2992M 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: Adjusting osd_memory_target on smithi172 to 3248M 2024-03-26T02:31:09.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:09.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:08 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi164 to 2992M 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:31:09.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: Adjusting osd_memory_target on smithi172 to 3248M 2024-03-26T02:31:09.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:09.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:09.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:08 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:09.829 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:09 smithi164 systemd[1]: Stopping Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:09.829 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:09 smithi164 bash[58637]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter.a" found: no such container 2024-03-26T02:31:09.829 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:09 smithi164 podman[58649]: 2024-03-26 02:31:09.738379306 +0000 UTC m=+0.158732795 container died 6fbe35cf8a99cb29336eda977758491b5a5d206276bc07c3e8f50e990aa05bf9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.095 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:31:10.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.097 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.097 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.097 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.097 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.097 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.097 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.097 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.097 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:09 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:09 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:10.347 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 podman[58649]: 2024-03-26 02:31:10.094410307 +0000 UTC m=+0.514763792 container cleanup 6fbe35cf8a99cb29336eda977758491b5a5d206276bc07c3e8f50e990aa05bf9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:10.347 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 bash[58649]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T02:31:10.347 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 podman[58663]: 2024-03-26 02:31:10.2955012 +0000 UTC m=+0.561223180 container remove 6fbe35cf8a99cb29336eda977758491b5a5d206276bc07c3e8f50e990aa05bf9 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:10.347 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:31:10.681 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 bash[58677]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter.a" found: no such container 2024-03-26T02:31:10.681 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T02:31:10.681 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 systemd[1]: Stopped Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:10.681 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 3.682s CPU time. 2024-03-26T02:31:10.681 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 systemd[1]: Starting Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:10.681 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 podman[58748]: 2024-03-26 02:31:10.478647573 +0000 UTC m=+0.017502268 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-26T02:31:10.682 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 podman[58748]: 2024-03-26 02:31:10.579818037 +0000 UTC m=+0.118672722 container create 547c321e8f73a49e95f2c6fa9b7524060cf48a7f30151907dc25f2d280d809de (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-mon[24851]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-26T02:31:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-mon[24851]: Reconfiguring daemon node-exporter.a on smithi164 2024-03-26T02:31:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-mon[24851]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:31:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-mon[29512]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-26T02:31:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-mon[29512]: Reconfiguring daemon node-exporter.a on smithi164 2024-03-26T02:31:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-mon[29512]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:31:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:11.045 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 podman[58748]: 2024-03-26 02:31:10.782905455 +0000 UTC m=+0.321760154 container init 547c321e8f73a49e95f2c6fa9b7524060cf48a7f30151907dc25f2d280d809de (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 podman[58748]: 2024-03-26 02:31:10.785674549 +0000 UTC m=+0.324529235 container start 547c321e8f73a49e95f2c6fa9b7524060cf48a7f30151907dc25f2d280d809de (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.789Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.789Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.789Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.789Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=arp 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-26T02:31:11.046 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=edac 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-26T02:31:11.047 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=os 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-26T02:31:11.048 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=stat 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=time 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=uname 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[58762]: ts=2024-03-26T02:31:10.790Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 bash[58748]: 547c321e8f73a49e95f2c6fa9b7524060cf48a7f30151907dc25f2d280d809de 2024-03-26T02:31:11.049 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:31:10 smithi164 systemd[1]: Started Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:10 smithi172 ceph-mon[25688]: Reconfiguring node-exporter.a (dependencies changed)... 2024-03-26T02:31:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:10 smithi172 ceph-mon[25688]: Reconfiguring daemon node-exporter.a on smithi164 2024-03-26T02:31:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:10 smithi172 ceph-mon[25688]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:31:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:10 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:31:11] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:31:12.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[29512]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[29512]: Reconfiguring daemon alertmanager.a on smithi164 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[29512]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[24851]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[24851]: Reconfiguring daemon alertmanager.a on smithi164 2024-03-26T02:31:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:11 smithi164 ceph-mon[24851]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:11 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:11 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:11 smithi172 ceph-mon[25688]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:31:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:11 smithi172 ceph-mon[25688]: Reconfiguring daemon alertmanager.a on smithi164 2024-03-26T02:31:12.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:11 smithi172 ceph-mon[25688]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:12.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:12.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.172:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.164:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:31:13.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:12 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:13.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:12 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:12 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:13.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=error ts=2024-03-26T02:31:13.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:31:13.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:31:13.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.164:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.164 because it doesn't contain any IP SANs" 2024-03-26T02:31:13.795 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=warn ts=2024-03-26T02:31:13.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.172:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.172 because it doesn't contain any IP SANs" 2024-03-26T02:31:14.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:14 smithi164 ceph-mon[24851]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:14.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:14 smithi164 ceph-mon[29512]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:14.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:14 smithi172 ceph-mon[25688]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:16.760 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:16 smithi164 ceph-mon[24851]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:16.761 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:16 smithi164 ceph-mon[29512]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:16 smithi172 ceph-mon[25688]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:31:18.289 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:18 smithi164 systemd[1]: Stopping Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:18.289 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:18 smithi164 bash[59022]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T02:31:18.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:31:17] "GET /metrics HTTP/1.1" 200 37907 "" "Prometheus/2.33.4" 2024-03-26T02:31:18.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:18 smithi164 ceph-mon[24851]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:18.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:18 smithi164 ceph-mon[29512]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:18.546 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[49453]: level=info ts=2024-03-26T02:31:18.288Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:31:18.546 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:18 smithi164 podman[59033]: 2024-03-26 02:31:18.360273901 +0000 UTC m=+0.178234263 container died 1fc3d79d0812a14cf4164c7b8c9071d6793b96fb8cb81efe5d02f999d3d5dfbc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:18.723 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:18 smithi172 ceph-mon[25688]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:18.866 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:18 smithi164 podman[59033]: 2024-03-26 02:31:18.840617387 +0000 UTC m=+0.658577740 container cleanup 1fc3d79d0812a14cf4164c7b8c9071d6793b96fb8cb81efe5d02f999d3d5dfbc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:18.866 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:18 smithi164 bash[59033]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:31:19.226 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:18 smithi164 bash[59060]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager.a" found: no such container 2024-03-26T02:31:19.226 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 podman[59047]: 2024-03-26 02:31:19.108693438 +0000 UTC m=+0.747814495 container remove 1fc3d79d0812a14cf4164c7b8c9071d6793b96fb8cb81efe5d02f999d3d5dfbc (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:19.226 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 podman[59047]: 2024-03-26 02:31:19.226010907 +0000 UTC m=+0.865131919 volume remove b619cd2ed5285507bb42c746c3b78f9bece4e83af8df2be8982e7a02b37dabad 2024-03-26T02:31:19.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:31:19.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 systemd[1]: Stopped Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:19.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.057s CPU time. 2024-03-26T02:31:19.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 systemd[1]: Starting Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:19.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 podman[59128]: 2024-03-26 02:31:19.362144095 +0000 UTC m=+0.018354844 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-03-26T02:31:19.545 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 podman[59128]: 2024-03-26 02:31:19.46029898 +0000 UTC m=+0.116509719 volume create 1ffc7feb0a4e9ee1cb8bda6ce5846783070ab9debae439e6e2885b55b07c51cd 2024-03-26T02:31:19.813 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 podman[59128]: 2024-03-26 02:31:19.560742258 +0000 UTC m=+0.216952999 container create 98dbe984702c222e227e51664578839258e1fb25c97ebdbc6c404612607f38fe (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:20.257 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 podman[59128]: 2024-03-26 02:31:19.813230034 +0000 UTC m=+0.469440773 container init 98dbe984702c222e227e51664578839258e1fb25c97ebdbc6c404612607f38fe (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:20.257 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 podman[59128]: 2024-03-26 02:31:19.816194205 +0000 UTC m=+0.472404942 container start 98dbe984702c222e227e51664578839258e1fb25c97ebdbc6c404612607f38fe (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:20.257 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:19.827Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-03-26T02:31:20.257 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:19.827Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-03-26T02:31:20.257 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:19.828Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.164 port=9094 2024-03-26T02:31:20.258 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:19.829Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T02:31:20.258 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:19.855Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:31:20.258 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:19.856Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:31:20.258 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:19.857Z caller=main.go:518 msg=Listening address=:9093 2024-03-26T02:31:20.258 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:19.858Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-03-26T02:31:20.258 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 bash[59128]: 98dbe984702c222e227e51664578839258e1fb25c97ebdbc6c404612607f38fe 2024-03-26T02:31:20.258 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:19 smithi164 systemd[1]: Started Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[24851]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[24851]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:31:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:31:20.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[29512]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[29512]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:31:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:31:20.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:20 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:20 smithi172 ceph-mon[25688]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:20 smithi172 ceph-mon[25688]: from='client.34634 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:20 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:20 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:20 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:31:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:20 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:31:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:20 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:21.782 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:21 smithi164 ceph-mon[24851]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:31:21.782 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:21 smithi164 ceph-mon[24851]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:31:21.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:21 smithi164 ceph-mon[29512]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:31:21.783 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:21 smithi164 ceph-mon[29512]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:31:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:21 smithi172 ceph-mon[25688]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:31:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:21 smithi172 ceph-mon[25688]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:31:22.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:31:21] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:31:22.045 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:21.830Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000087823s 2024-03-26T02:31:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:22 smithi164 ceph-mon[24851]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:22.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:22 smithi164 ceph-mon[29512]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:22.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:22 smithi172 ceph-mon[25688]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:22.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.172:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.164:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:31:24.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:24 smithi164 ceph-mon[24851]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:24.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:24 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:24.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:24 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:24.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:24 smithi164 ceph-mon[29512]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:24.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:24 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:24.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:24 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:24.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:24 smithi172 ceph-mon[25688]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:24 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:24 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:25.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:25 smithi164 ceph-mon[24851]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:31:25.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:25 smithi164 ceph-mon[24851]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:31:25.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:25 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1711331971' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:31:25.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:25 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/4124252908' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3783252420"}]: dispatch 2024-03-26T02:31:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:25 smithi164 ceph-mon[29512]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:31:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:25 smithi164 ceph-mon[29512]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:31:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:25 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1711331971' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:31:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:25 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/4124252908' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3783252420"}]: dispatch 2024-03-26T02:31:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:25 smithi172 ceph-mon[25688]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:31:25.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:25 smithi172 ceph-mon[25688]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:31:25.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:25 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1711331971' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:31:25.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:25 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/4124252908' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3783252420"}]: dispatch 2024-03-26T02:31:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:26 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/4124252908' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3783252420"}]': finished 2024-03-26T02:31:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:26 smithi164 ceph-mon[24851]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T02:31:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:26 smithi164 ceph-mon[24851]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T02:31:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:26 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2096680322' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2071275137"}]: dispatch 2024-03-26T02:31:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:26 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/4124252908' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3783252420"}]': finished 2024-03-26T02:31:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:26 smithi164 ceph-mon[29512]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T02:31:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:26 smithi164 ceph-mon[29512]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T02:31:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:26 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2096680322' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2071275137"}]: dispatch 2024-03-26T02:31:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:26 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/4124252908' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3783252420"}]': finished 2024-03-26T02:31:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:26 smithi172 ceph-mon[25688]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T02:31:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:26 smithi172 ceph-mon[25688]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T02:31:26.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:26 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2096680322' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2071275137"}]: dispatch 2024-03-26T02:31:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:27 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2096680322' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2071275137"}]': finished 2024-03-26T02:31:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:27 smithi164 ceph-mon[24851]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T02:31:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:27 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2549819205' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]: dispatch 2024-03-26T02:31:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:27 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]: dispatch 2024-03-26T02:31:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:27 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2096680322' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2071275137"}]': finished 2024-03-26T02:31:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:27 smithi164 ceph-mon[29512]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T02:31:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:27 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2549819205' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]: dispatch 2024-03-26T02:31:27.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:27 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]: dispatch 2024-03-26T02:31:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:27 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2096680322' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/2071275137"}]': finished 2024-03-26T02:31:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:27 smithi172 ceph-mon[25688]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T02:31:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:27 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2549819205' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]: dispatch 2024-03-26T02:31:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:27 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]: dispatch 2024-03-26T02:31:28.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:31:27] "GET /metrics HTTP/1.1" 200 37899 "" "Prometheus/2.33.4" 2024-03-26T02:31:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]': finished 2024-03-26T02:31:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[24851]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T02:31:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[24851]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:31:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/38784012' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3230998345"}]: dispatch 2024-03-26T02:31:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]': finished 2024-03-26T02:31:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[29512]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T02:31:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[29512]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:31:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:28 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/38784012' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3230998345"}]: dispatch 2024-03-26T02:31:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:28 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/2071275137"}]': finished 2024-03-26T02:31:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:28 smithi172 ceph-mon[25688]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T02:31:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:28 smithi172 ceph-mon[25688]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:31:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:28 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:28 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/38784012' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3230998345"}]: dispatch 2024-03-26T02:31:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:29 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/38784012' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3230998345"}]': finished 2024-03-26T02:31:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:29 smithi164 ceph-mon[24851]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T02:31:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:29 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/526217623' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1259649196"}]: dispatch 2024-03-26T02:31:29.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:29 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/38784012' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3230998345"}]': finished 2024-03-26T02:31:29.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:29 smithi164 ceph-mon[29512]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T02:31:29.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:29 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/526217623' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1259649196"}]: dispatch 2024-03-26T02:31:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:29 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/38784012' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3230998345"}]': finished 2024-03-26T02:31:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:29 smithi172 ceph-mon[25688]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T02:31:29.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:29 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/526217623' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1259649196"}]: dispatch 2024-03-26T02:31:30.294 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:31:29 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:31:29.831Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001930582s 2024-03-26T02:31:30.672 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:30 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/526217623' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1259649196"}]': finished 2024-03-26T02:31:30.672 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:30 smithi164 ceph-mon[29512]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:30.672 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:30 smithi164 ceph-mon[29512]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T02:31:30.672 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:30 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1567702775' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/778903593"}]: dispatch 2024-03-26T02:31:30.672 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:30 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/526217623' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1259649196"}]': finished 2024-03-26T02:31:30.672 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:30 smithi164 ceph-mon[24851]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:30.672 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:30 smithi164 ceph-mon[24851]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T02:31:30.672 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:30 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1567702775' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/778903593"}]: dispatch 2024-03-26T02:31:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:30 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/526217623' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1259649196"}]': finished 2024-03-26T02:31:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:30 smithi172 ceph-mon[25688]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:30 smithi172 ceph-mon[25688]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T02:31:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:30 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1567702775' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/778903593"}]: dispatch 2024-03-26T02:31:31.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:31 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1567702775' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/778903593"}]': finished 2024-03-26T02:31:31.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:31 smithi164 ceph-mon[24851]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T02:31:31.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:31 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/601527729' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/258825764"}]: dispatch 2024-03-26T02:31:31.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:31 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1567702775' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/778903593"}]': finished 2024-03-26T02:31:31.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:31 smithi164 ceph-mon[29512]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T02:31:31.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:31 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/601527729' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/258825764"}]: dispatch 2024-03-26T02:31:31.681 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:31 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1567702775' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/778903593"}]': finished 2024-03-26T02:31:31.682 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:31 smithi172 ceph-mon[25688]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T02:31:31.682 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:31 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/601527729' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/258825764"}]: dispatch 2024-03-26T02:31:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:31:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[25087]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:31:31] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:31:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:32 smithi164 ceph-mon[24851]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:32 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/601527729' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/258825764"}]': finished 2024-03-26T02:31:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:32 smithi164 ceph-mon[24851]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T02:31:32.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:32 smithi164 ceph-mon[29512]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:32.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:32 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/601527729' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/258825764"}]': finished 2024-03-26T02:31:32.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:32 smithi164 ceph-mon[29512]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T02:31:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:32 smithi172 ceph-mon[25688]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:32 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/601527729' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/258825764"}]': finished 2024-03-26T02:31:32.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:32 smithi172 ceph-mon[25688]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T02:31:32.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:32 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:32.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.172:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.164:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:31:34.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-mon[25688]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T02:31:34.545 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 systemd[1]: Stopping Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:34.545 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 bash[49108]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:31:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:34 smithi164 ceph-mon[24851]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T02:31:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:34 smithi164 ceph-mon[29512]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 81 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:31:34.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T02:31:34.808 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[42620]: ts=2024-03-26T02:31:34.545Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T02:31:34.808 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:34 smithi172 podman[49121]: 2024-03-26 02:31:34.55538931 +0000 UTC m=+0.127622226 container died cc5b561b62662704c658e9bad59cdfee4f4161087951003ae5eb31b8729d5df0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:35.743 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:35 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:35.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:35 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:35.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:35 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:36.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:35 smithi172 podman[49121]: 2024-03-26 02:31:35.815738906 +0000 UTC m=+1.387971817 container cleanup cc5b561b62662704c658e9bad59cdfee4f4161087951003ae5eb31b8729d5df0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:36.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:35 smithi172 bash[49121]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:31:36.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:35 smithi172 bash[49148]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:31:36.351 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 podman[49135]: 2024-03-26 02:31:36.100567974 +0000 UTC m=+1.548166465 container remove cc5b561b62662704c658e9bad59cdfee4f4161087951003ae5eb31b8729d5df0 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:36.351 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T02:31:36.351 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 systemd[1]: Stopped Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:36.351 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.979s CPU time. 2024-03-26T02:31:36.351 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 systemd[1]: Starting Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:36.352 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 podman[49218]: 2024-03-26 02:31:36.239553672 +0000 UTC m=+0.018979788 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T02:31:36.352 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 podman[49218]: 2024-03-26 02:31:36.351182719 +0000 UTC m=+0.130608819 container create 7584c8ee6d15c843a1ca80ac473c064c5259bbd0c2eeadd41b72a9ebe9b94374 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:36.601 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-mon[25688]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:31:36.601 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 podman[49218]: 2024-03-26 02:31:36.57003813 +0000 UTC m=+0.349464231 container init 7584c8ee6d15c843a1ca80ac473c064c5259bbd0c2eeadd41b72a9ebe9b94374 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 podman[49218]: 2024-03-26 02:31:36.572522654 +0000 UTC m=+0.351948758 container start 7584c8ee6d15c843a1ca80ac473c064c5259bbd0c2eeadd41b72a9ebe9b94374 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.595Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.595Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.595Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.595Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi172 (none))" 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.595Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.595Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.597Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.598Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.599Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.601Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:31:36.602 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.601Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.876µs 2024-03-26T02:31:36.603 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.601Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:31:36.695 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:36 smithi164 ceph-mon[29512]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:31:36.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:36 smithi164 ceph-mon[24851]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:31:36.918 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 bash[49218]: 7584c8ee6d15c843a1ca80ac473c064c5259bbd0c2eeadd41b72a9ebe9b94374 2024-03-26T02:31:36.918 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 systemd[1]: Started Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:36.918 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:36.816Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-03-26T02:31:37.285 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:37.093Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-03-26T02:31:37.285 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:37.093Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-03-26T02:31:37.285 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:37.093Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.794µs wal_replay_duration=492.367254ms total_replay_duration=492.411197ms 2024-03-26T02:31:37.285 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:37.096Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:31:37.286 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:37.096Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T02:31:37.286 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:37.097Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:31:37.286 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:37.121Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.902983ms db_storage=957ns remote_storage=1.485µs web_handler=484ns query_engine=895ns scrape=476.797µs scrape_sd=105.117µs notify=15.928µs notify_sd=37.647µs rules=23.845241ms 2024-03-26T02:31:37.286 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:37.121Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T02:31:37.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:37 smithi172 systemd[1]: Stopping Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:37.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:37 smithi172 bash[49351]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter.b" found: no such container 2024-03-26T02:31:37.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:37 smithi172 podman[49363]: 2024-03-26 02:31:37.417604995 +0000 UTC m=+0.100723626 container died 9c5963ac47a19945674c24a86f0c42ff0d5da17d6694da43d3ddf29f8aff3fc7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:31:37.983 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:37.983 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:37.983 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-mon[25688]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-26T02:31:37.983 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:37 smithi172 ceph-mon[25688]: Reconfiguring daemon node-exporter.b on smithi172 2024-03-26T02:31:37.984 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:37 smithi172 podman[49363]: 2024-03-26 02:31:37.748520189 +0000 UTC m=+0.431638826 container cleanup 9c5963ac47a19945674c24a86f0c42ff0d5da17d6694da43d3ddf29f8aff3fc7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:31:37.984 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:37 smithi172 bash[49363]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T02:31:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:37 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:37 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:37 smithi164 ceph-mon[24851]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-26T02:31:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:37 smithi164 ceph-mon[24851]: Reconfiguring daemon node-exporter.b on smithi172 2024-03-26T02:31:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:37 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:37 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:37 smithi164 ceph-mon[29512]: Reconfiguring node-exporter.b (dependencies changed)... 2024-03-26T02:31:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:37 smithi164 ceph-mon[29512]: Reconfiguring daemon node-exporter.b on smithi172 2024-03-26T02:31:38.235 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:37 smithi172 podman[49379]: 2024-03-26 02:31:37.983239103 +0000 UTC m=+0.569665732 container remove 9c5963ac47a19945674c24a86f0c42ff0d5da17d6694da43d3ddf29f8aff3fc7 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:31:38.236 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:37 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:31:38.236 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 bash[49393]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter.b" found: no such container 2024-03-26T02:31:38.236 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T02:31:38.236 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 systemd[1]: Stopped Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:38.236 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 3.834s CPU time. 2024-03-26T02:31:38.236 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 systemd[1]: Starting Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:38.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 podman[49463]: 2024-03-26 02:31:38.137708387 +0000 UTC m=+0.017611748 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-03-26T02:31:38.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 podman[49463]: 2024-03-26 02:31:38.267447011 +0000 UTC m=+0.147350360 container create 114ffb73b2787362f22fc081496c30c5e7d085de66cd2d63504e92d830aaebce (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:31:38.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 podman[49463]: 2024-03-26 02:31:38.469786717 +0000 UTC m=+0.349690066 container init 114ffb73b2787362f22fc081496c30c5e7d085de66cd2d63504e92d830aaebce (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:31:38.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 podman[49463]: 2024-03-26 02:31:38.472415239 +0000 UTC m=+0.352318597 container start 114ffb73b2787362f22fc081496c30c5e7d085de66cd2d63504e92d830aaebce (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.476Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.476Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.476Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.476Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=arp 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=bcache 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=bonding 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=btrfs 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=conntrack 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=cpu 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-03-26T02:31:38.558 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=diskstats 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=dmi 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=edac 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=entropy 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=filefd 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=filesystem 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=hwmon 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=infiniband 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=ipvs 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=loadavg 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=mdadm 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=meminfo 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=netclass 2024-03-26T02:31:38.559 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=netdev 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=netstat 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=nfs 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=nfsd 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=nvme 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=os 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=pressure 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=rapl 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=schedstat 2024-03-26T02:31:38.560 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=sockstat 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=softnet 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=stat 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=tapestats 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=textfile 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=time 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=uname 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=vmstat 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=xfs 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:115 level=info collector=zfs 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-03-26T02:31:38.561 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[49478]: ts=2024-03-26T02:31:38.477Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-03-26T02:31:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:38 smithi164 ceph-mon[24851]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:31:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:38 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:38 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:38 smithi164 ceph-mon[29512]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:31:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:38 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:38 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:39.056 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 bash[49463]: 114ffb73b2787362f22fc081496c30c5e7d085de66cd2d63504e92d830aaebce 2024-03-26T02:31:39.056 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:31:38 smithi172 systemd[1]: Started Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-mon[25688]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:31:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:38 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[24851]: Reconfiguring grafana.a (dependencies changed)... 2024-03-26T02:31:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[24851]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:31:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[24851]: Reconfiguring daemon grafana.a on smithi172 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[29512]: Reconfiguring grafana.a (dependencies changed)... 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[29512]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:31:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:39 smithi164 ceph-mon[29512]: Reconfiguring daemon grafana.a on smithi172 2024-03-26T02:31:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:39 smithi172 ceph-mon[25688]: Reconfiguring grafana.a (dependencies changed)... 2024-03-26T02:31:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:39 smithi172 ceph-mon[25688]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:31:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:39 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:39 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:39 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:31:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:39 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:31:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:39 smithi172 ceph-mon[25688]: Reconfiguring daemon grafana.a on smithi172 2024-03-26T02:31:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:40 smithi164 ceph-mon[24851]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-26T02:31:41.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:40 smithi164 ceph-mon[29512]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-26T02:31:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:40 smithi172 ceph-mon[25688]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2024-03-26T02:31:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:42 smithi164 ceph-mon[24851]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T02:31:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:42 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:42 smithi164 ceph-mon[29512]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T02:31:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:42 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:42 smithi172 ceph-mon[25688]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T02:31:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:42 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:44 smithi164 ceph-mon[24851]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s 2024-03-26T02:31:44.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:44 smithi164 ceph-mon[29512]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s 2024-03-26T02:31:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:44 smithi172 ceph-mon[25688]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 0 op/s 2024-03-26T02:31:45.714 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:45 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:45.715 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:31:45] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-03-26T02:31:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:45 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:45.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:45 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:46.632 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:46 smithi172 ceph-mon[25688]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:46 smithi164 ceph-mon[24851]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:46.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:46 smithi164 ceph-mon[29512]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:48.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:47 smithi172 systemd[1]: Stopping Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:48.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:47 smithi172 bash[49758]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana.a" found: no such container 2024-03-26T02:31:48.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:31:47+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-03-26T02:31:48.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[41886]: t=2024-03-26T02:31:47+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-03-26T02:31:48.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:47 smithi172 podman[49771]: 2024-03-26 02:31:47.969112267 +0000 UTC m=+0.138132519 container died dfc5e042a9b59bab308b8a3b3b8d6042e3bf721640cbad2f063bcccb9245a927 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.24.2, version=8.5, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , io.openshift.expose-services=, name=ubi8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container) 2024-03-26T02:31:48.659 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:48 smithi172 ceph-mon[25688]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:48.660 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 podman[49771]: 2024-03-26 02:31:48.389430751 +0000 UTC m=+0.558451010 container cleanup dfc5e042a9b59bab308b8a3b3b8d6042e3bf721640cbad2f063bcccb9245a927 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, vendor=Red Hat, Inc., distribution-scope=public, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.openshift.expose-services=) 2024-03-26T02:31:48.660 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 bash[49771]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a 2024-03-26T02:31:48.660 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 bash[49800]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana.a" found: no such container 2024-03-26T02:31:48.660 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 podman[49785]: 2024-03-26 02:31:48.615642183 +0000 UTC m=+0.646731095 container remove dfc5e042a9b59bab308b8a3b3b8d6042e3bf721640cbad2f063bcccb9245a927 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, distribution-scope=public, version=8.5, description=Ceph Grafana Container) 2024-03-26T02:31:48.660 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-26T02:31:48.660 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 systemd[1]: Stopped Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:48.660 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Consumed 3.363s CPU time. 2024-03-26T02:31:48.660 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 systemd[1]: Starting Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:31:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:48 smithi164 ceph-mon[24851]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:48.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:48 smithi164 ceph-mon[29512]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:48.980 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:31:48.980 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 podman[49868]: 2024-03-26 02:31:48.74813709 +0000 UTC m=+0.017792851 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-26T02:31:48.980 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:48 smithi172 podman[49868]: 2024-03-26 02:31:48.875040817 +0000 UTC m=+0.144696618 container create 98bea5d4b50cb48d7562645823ffc6e4f3c11d9edd154032391f1263f666582e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, vcs-type=git, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, name=ubi8) 2024-03-26T02:31:49.411 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STOPPING 2024-03-26T02:31:49.412 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:31:49.412 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STOPPED 2024-03-26T02:31:49.412 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STARTING 2024-03-26T02:31:49.412 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Serving on http://:::9283 2024-03-26T02:31:49.412 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 podman[49868]: 2024-03-26 02:31:49.102599528 +0000 UTC m=+0.372255286 container init 98bea5d4b50cb48d7562645823ffc6e4f3c11d9edd154032391f1263f666582e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, version=8.5, description=Ceph Grafana Container, io.buildah.version=1.24.2, architecture=x86_64, vendor=Red Hat, Inc., name=ubi8, maintainer=Paul Cuzner ) 2024-03-26T02:31:49.412 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 podman[49868]: 2024-03-26 02:31:49.105215695 +0000 UTC m=+0.374871457 container start 98bea5d4b50cb48d7562645823ffc6e4f3c11d9edd154032391f1263f666582e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, vendor=Red Hat, Inc., description=Ceph Grafana Container, architecture=x86_64, vcs-type=git, distribution-scope=public, name=ubi8, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182) 2024-03-26T02:31:49.412 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[49882]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:31:49.412 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[49882]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:31:49.412 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 bash[49868]: 98bea5d4b50cb48d7562645823ffc6e4f3c11d9edd154032391f1263f666582e 2024-03-26T02:31:49.412 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 systemd[1]: Started Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:31:49.413 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 podman[49899]: 2024-03-26 02:31:49.236003033 +0000 UTC m=+0.097471728 container died 98bea5d4b50cb48d7562645823ffc6e4f3c11d9edd154032391f1263f666582e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, maintainer=Paul Cuzner , vendor=Red Hat, Inc., architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, description=Ceph Grafana Container, name=ubi8, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-26T02:31:49.767 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STARTED 2024-03-26T02:31:49.767 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STOPPING 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STOPPED 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STARTING 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Serving on http://:::9283 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STARTED 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STOPPING 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STOPPED 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STARTING 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Serving on http://:::9283 2024-03-26T02:31:49.768 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:31:49] ENGINE Bus STARTED 2024-03-26T02:31:50.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 podman[49899]: 2024-03-26 02:31:49.831799947 +0000 UTC m=+0.693268629 container remove 98bea5d4b50cb48d7562645823ffc6e4f3c11d9edd154032391f1263f666582e (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, vcs-type=git, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, description=Ceph Grafana Container, name=ubi8, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.expose-services=, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner ) 2024-03-26T02:31:50.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-26T02:31:50.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:49 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi164.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi164.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:31:50.405 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.406 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[24851]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi164.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi164.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:31:50.407 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:31:50.408 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:50 smithi164 ceph-mon[29512]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:50.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:31:50.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:31:50.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi164.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:31:50.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi164.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:31:50.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi172.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:31:50.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:31:50.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:31:50.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:31:50.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi172.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:31:50.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:50.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:31:50.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:50 smithi172 ceph-mon[25688]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:52 smithi164 ceph-mon[24851]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:52 smithi164 ceph-mon[29512]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:52 smithi172 ceph-mon[25688]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:53.806 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:31:54.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:54.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:54.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:54.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:31:54.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:53 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:31:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:53 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:55.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:54 smithi164 ceph-mon[24851]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:31:55.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:54 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:55.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:54 smithi164 ceph-mon[29512]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:31:55.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:54 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:55.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:54 smithi172 ceph-mon[25688]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:31:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:54 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:31:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:31:55] "GET /metrics HTTP/1.1" 200 37903 "" "Prometheus/2.33.4" 2024-03-26T02:31:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:55 smithi164 ceph-mon[24851]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:31:56.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:55 smithi164 ceph-mon[29512]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:31:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:55 smithi172 ceph-mon[25688]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:31:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:58 smithi164 ceph-mon[24851]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:31:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:58.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:31:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:58 smithi164 ceph-mon[29512]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:31:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:31:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:58.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:58 smithi172 ceph-mon[25688]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:31:58.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:31:58.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:31:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:58.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:31:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:31:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:00.155 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:59 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 1. 2024-03-26T02:32:00.155 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:59 smithi172 systemd[1]: Stopped Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:00.155 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:31:59 smithi172 systemd[1]: Starting Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:32:00.155 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 podman[50565]: 2024-03-26 02:32:00.045421376 +0000 UTC m=+0.017563652 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-26T02:32:00.407 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 podman[50565]: 2024-03-26 02:32:00.155125462 +0000 UTC m=+0.127267766 container create 1c9a73dde970b842048b61c08cd472151f4fc74d16f3af494c396ec461a41ba6 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, vcs-type=git, architecture=x86_64, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, release=236.1648460182, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, name=ubi8, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc.) 2024-03-26T02:32:00.408 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 podman[50565]: 2024-03-26 02:32:00.407681636 +0000 UTC m=+0.379823898 container init 1c9a73dde970b842048b61c08cd472151f4fc74d16f3af494c396ec461a41ba6 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.buildah.version=1.24.2, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, release=236.1648460182, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.openshift.tags=base rhel8, description=Ceph Grafana Container) 2024-03-26T02:32:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:00 smithi164 ceph-mon[24851]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:00.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:00 smithi164 ceph-mon[29512]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:00 smithi172 ceph-mon[25688]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:00.806 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 podman[50565]: 2024-03-26 02:32:00.410408663 +0000 UTC m=+0.382550926 container start 1c9a73dde970b842048b61c08cd472151f4fc74d16f3af494c396ec461a41ba6 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, vcs-type=git, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, distribution-scope=public, architecture=x86_64, build-date=2022-03-28T10:36:18.413762) 2024-03-26T02:32:00.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[50580]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:32:00.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[50580]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:32:00.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 bash[50565]: 1c9a73dde970b842048b61c08cd472151f4fc74d16f3af494c396ec461a41ba6 2024-03-26T02:32:00.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 systemd[1]: Started Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:00.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:00 smithi172 podman[50596]: 2024-03-26 02:32:00.516168256 +0000 UTC m=+0.073026623 container died 1c9a73dde970b842048b61c08cd472151f4fc74d16f3af494c396ec461a41ba6 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, maintainer=Paul Cuzner , io.buildah.version=1.24.2, architecture=x86_64, io.openshift.tags=base rhel8, release=236.1648460182, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.component=ubi8-container, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5) 2024-03-26T02:32:02.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:01 smithi172 podman[50596]: 2024-03-26 02:32:01.720089768 +0000 UTC m=+1.276948180 container remove 1c9a73dde970b842048b61c08cd472151f4fc74d16f3af494c396ec461a41ba6 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, maintainer=Paul Cuzner , name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., release=236.1648460182, vcs-type=git) 2024-03-26T02:32:02.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:01 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-26T02:32:02.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:01 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:32:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:02 smithi164 ceph-mon[24851]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:02 smithi164 ceph-mon[29512]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:02 smithi172 ceph-mon[25688]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:03.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:04 smithi164 ceph-mon[24851]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:04 smithi164 ceph-mon[29512]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:04 smithi172 ceph-mon[25688]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1019 B/s rd, 0 op/s 2024-03-26T02:32:05.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:05 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:05.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:05 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:05.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:05 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:05.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:32:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:32:05] "GET /metrics HTTP/1.1" 200 37977 "" "Prometheus/2.33.4" 2024-03-26T02:32:06.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:06 smithi164 ceph-mon[29512]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:06.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:06 smithi164 ceph-mon[24851]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:06 smithi172 ceph-mon[25688]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:08 smithi164 ceph-mon[24851]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:08.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:08 smithi164 ceph-mon[29512]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:08.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:08 smithi172 ceph-mon[25688]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:08.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:08 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:10 smithi164 ceph-mon[24851]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:10.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:10 smithi164 ceph-mon[29512]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:10.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:10 smithi172 ceph-mon[25688]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:12.295 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:11 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 2. 2024-03-26T02:32:12.295 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:11 smithi172 systemd[1]: Stopped Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:12.295 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:11 smithi172 systemd[1]: Starting Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:32:12.295 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 podman[50665]: 2024-03-26 02:32:11.941888917 +0000 UTC m=+0.017504392 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-26T02:32:12.295 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 podman[50665]: 2024-03-26 02:32:12.053422291 +0000 UTC m=+0.129037781 container create 8465befcc376ec8ee915497773c14abbdfc24a26a7ceafa018433d2b56b3db00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, version=8.5, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , io.buildah.version=1.24.2, com.redhat.component=ubi8-container, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.openshift.tags=base rhel8, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-26T02:32:12.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:12 smithi172 ceph-mon[25688]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:12.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:12 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:12.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 podman[50665]: 2024-03-26 02:32:12.39231168 +0000 UTC m=+0.467927128 container init 8465befcc376ec8ee915497773c14abbdfc24a26a7ceafa018433d2b56b3db00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , name=ubi8, io.buildah.version=1.24.2, io.openshift.expose-services=, description=Ceph Grafana Container, release=236.1648460182, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, vcs-type=git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-26T02:32:12.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 podman[50665]: 2024-03-26 02:32:12.39515978 +0000 UTC m=+0.470775228 container start 8465befcc376ec8ee915497773c14abbdfc24a26a7ceafa018433d2b56b3db00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.component=ubi8-container, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, name=ubi8, io.openshift.expose-services=, description=Ceph Grafana Container, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-03-26T02:32:12.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[50680]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:32:12.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[50680]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:32:12.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 bash[50665]: 8465befcc376ec8ee915497773c14abbdfc24a26a7ceafa018433d2b56b3db00 2024-03-26T02:32:12.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 systemd[1]: Started Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:12.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:12 smithi172 podman[50697]: 2024-03-26 02:32:12.508572487 +0000 UTC m=+0.080187240 container died 8465befcc376ec8ee915497773c14abbdfc24a26a7ceafa018433d2b56b3db00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, distribution-scope=public, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, release=236.1648460182, name=ubi8, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762) 2024-03-26T02:32:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:12 smithi164 ceph-mon[24851]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:12 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:12 smithi164 ceph-mon[29512]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:12 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:13.462 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:13 smithi172 podman[50697]: 2024-03-26 02:32:13.070802574 +0000 UTC m=+0.642417332 container remove 8465befcc376ec8ee915497773c14abbdfc24a26a7ceafa018433d2b56b3db00 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.openshift.expose-services=, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, version=8.5, name=ubi8, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, maintainer=Paul Cuzner , architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container) 2024-03-26T02:32:13.462 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:13 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-26T02:32:13.463 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:13 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:32:13.806 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:14.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:14 smithi164 ceph-mon[24851]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:14.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:14 smithi164 ceph-mon[29512]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:14.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:14 smithi172 ceph-mon[25688]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:15.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:15 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:15.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:15 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:15 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:15.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:32:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:32:15] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-26T02:32:16.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:16 smithi164 ceph-mon[24851]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:16 smithi164 ceph-mon[29512]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:16 smithi172 ceph-mon[25688]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:18.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:18 smithi164 ceph-mon[24851]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:18.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:18 smithi164 ceph-mon[29512]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:18 smithi172 ceph-mon[25688]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:18.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:20 smithi164 ceph-mon[24851]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:20 smithi164 ceph-mon[29512]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:20 smithi172 ceph-mon[25688]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:22 smithi164 ceph-mon[24851]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:22 smithi164 ceph-mon[29512]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:22 smithi172 ceph-mon[25688]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:23.666 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:23.667 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 3. 2024-03-26T02:32:23.667 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 systemd[1]: Stopped Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:23.667 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 systemd[1]: Starting Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:32:23.667 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 podman[50774]: 2024-03-26 02:32:23.451053009 +0000 UTC m=+0.017694037 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-26T02:32:23.667 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 podman[50774]: 2024-03-26 02:32:23.556821895 +0000 UTC m=+0.123462896 container create 5005594f0353230cc3b5de2bccb37b91e2bce52fd0e16e51a7b4b53a572dea56 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, io.buildah.version=1.24.2, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.openshift.expose-services=, distribution-scope=public, name=ubi8, vcs-type=git, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762) 2024-03-26T02:32:24.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 podman[50774]: 2024-03-26 02:32:23.767462603 +0000 UTC m=+0.334103615 container init 5005594f0353230cc3b5de2bccb37b91e2bce52fd0e16e51a7b4b53a572dea56 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, name=ubi8, architecture=x86_64, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, io.buildah.version=1.24.2, vendor=Red Hat, Inc., maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-03-26T02:32:24.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 podman[50774]: 2024-03-26 02:32:23.770028014 +0000 UTC m=+0.336669017 container start 5005594f0353230cc3b5de2bccb37b91e2bce52fd0e16e51a7b4b53a572dea56 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, distribution-scope=public, vcs-type=git, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, version=8.5, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , io.openshift.expose-services=) 2024-03-26T02:32:24.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[50790]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:32:24.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[50790]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:32:24.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 bash[50774]: 5005594f0353230cc3b5de2bccb37b91e2bce52fd0e16e51a7b4b53a572dea56 2024-03-26T02:32:24.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 systemd[1]: Started Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:24.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:23 smithi172 podman[50806]: 2024-03-26 02:32:23.950197125 +0000 UTC m=+0.147510747 container died 5005594f0353230cc3b5de2bccb37b91e2bce52fd0e16e51a7b4b53a572dea56 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, name=ubi8, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, version=8.5, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vcs-type=git, maintainer=Paul Cuzner , description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762) 2024-03-26T02:32:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:24 smithi164 ceph-mon[24851]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:24 smithi164 ceph-mon[29512]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:24 smithi172 ceph-mon[25688]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:24.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:24 smithi172 podman[50806]: 2024-03-26 02:32:24.556387419 +0000 UTC m=+0.753701083 container remove 5005594f0353230cc3b5de2bccb37b91e2bce52fd0e16e51a7b4b53a572dea56 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, name=ubi8, com.redhat.component=ubi8-container, release=236.1648460182, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, vcs-type=git, description=Ceph Grafana Container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , vendor=Red Hat, Inc., distribution-scope=public, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-26T02:32:24.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:24 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-26T02:32:24.808 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:24 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:32:25.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:25 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:25.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:25 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:25 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:25.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:32:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:32:25] "GET /metrics HTTP/1.1" 200 37973 "" "Prometheus/2.33.4" 2024-03-26T02:32:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:26 smithi164 ceph-mon[24851]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:26.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:26 smithi164 ceph-mon[29512]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:26 smithi172 ceph-mon[25688]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:27 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:27 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:28.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:27 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:28 smithi164 ceph-mon[24851]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:29.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:28 smithi164 ceph-mon[29512]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:29.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:28 smithi172 ceph-mon[25688]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:29.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:30 smithi164 ceph-mon[24851]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:30 smithi164 ceph-mon[29512]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:30 smithi172 ceph-mon[25688]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:32 smithi164 ceph-mon[24851]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:33.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:32 smithi164 ceph-mon[29512]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:33.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:32 smithi172 ceph-mon[25688]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:33.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:34 smithi164 ceph-mon[24851]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:34 smithi164 ceph-mon[29512]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:34 smithi172 ceph-mon[25688]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:35.161 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:34 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 4. 2024-03-26T02:32:35.161 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:34 smithi172 systemd[1]: Stopped Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:35.161 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:34 smithi172 systemd[1]: Starting Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:32:35.161 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 podman[50876]: 2024-03-26 02:32:34.938546912 +0000 UTC m=+0.017769206 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-03-26T02:32:35.161 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 podman[50876]: 2024-03-26 02:32:35.056453079 +0000 UTC m=+0.135675428 container create 462f9d1d4a43ed62223a8aac1bf001411f1e6da8d80e157e08b52c8dd7c28311 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, description=Ceph Grafana Container, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-26T02:32:35.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:35 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:35.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:32:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:32:35] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-03-26T02:32:35.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 podman[50876]: 2024-03-26 02:32:35.275497784 +0000 UTC m=+0.354720079 container init 462f9d1d4a43ed62223a8aac1bf001411f1e6da8d80e157e08b52c8dd7c28311 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, distribution-scope=public, architecture=x86_64, io.openshift.tags=base rhel8, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, vcs-type=git, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, name=ubi8, vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container) 2024-03-26T02:32:35.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 podman[50876]: 2024-03-26 02:32:35.278187312 +0000 UTC m=+0.357409612 container start 462f9d1d4a43ed62223a8aac1bf001411f1e6da8d80e157e08b52c8dd7c28311 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, build-date=2022-03-28T10:36:18.413762, description=Ceph Grafana Container, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , name=ubi8, io.openshift.expose-services=, architecture=x86_64, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., version=8.5, io.openshift.tags=base rhel8) 2024-03-26T02:32:35.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[50891]: Failed to start grafana. error: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:32:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[50891]: failed to create log directory "/usr/share/grafana/data/log": mkdir /usr/share/grafana/data: permission denied 2024-03-26T02:32:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 bash[50876]: 462f9d1d4a43ed62223a8aac1bf001411f1e6da8d80e157e08b52c8dd7c28311 2024-03-26T02:32:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 systemd[1]: Started Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 podman[50907]: 2024-03-26 02:32:35.382595568 +0000 UTC m=+0.072153201 container died 462f9d1d4a43ed62223a8aac1bf001411f1e6da8d80e157e08b52c8dd7c28311 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, description=Ceph Grafana Container, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , vcs-type=git, io.openshift.expose-services=, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-03-26T02:32:35.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:35 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:35.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:35 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:35.921 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 podman[50907]: 2024-03-26 02:32:35.903995501 +0000 UTC m=+0.593553122 container remove 462f9d1d4a43ed62223a8aac1bf001411f1e6da8d80e157e08b52c8dd7c28311 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, io.openshift.expose-services=, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , release=236.1648460182, vcs-type=git, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., name=ubi8) 2024-03-26T02:32:35.922 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=1/FAILURE 2024-03-26T02:32:35.922 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:35 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:32:36.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:36 smithi164 ceph-mon[24851]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:36.696 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:36 smithi164 ceph-mon[29512]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:36 smithi172 ceph-mon[25688]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:38 smithi164 ceph-mon[24851]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:38.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:38 smithi164 ceph-mon[29512]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:38.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:38 smithi172 ceph-mon[25688]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:38.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:40 smithi164 ceph-mon[24851]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:40 smithi164 ceph-mon[29512]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:40 smithi172 ceph-mon[25688]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:42.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:42 smithi164 ceph-mon[29512]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:42.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:42 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:42 smithi164 ceph-mon[24851]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:42.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:42 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:42.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:42 smithi172 ceph-mon[25688]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:42.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:42 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:43.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:44 smithi164 ceph-mon[24851]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:44.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:44 smithi164 ceph-mon[29512]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:44 smithi172 ceph-mon[25688]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:45 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:45.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:45 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:45.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:45 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:45.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:32:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:32:45] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-26T02:32:46.400 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:46 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Scheduled restart job, restart counter is at 5. 2024-03-26T02:32:46.400 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:46 smithi172 systemd[1]: Stopped Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:46.400 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:46 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Start request repeated too quickly. 2024-03-26T02:32:46.400 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:46 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:32:46.400 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:32:46 smithi172 systemd[1]: Failed to start Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:32:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:46 smithi164 ceph-mon[24851]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:46.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:46 smithi164 ceph-mon[29512]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:46 smithi172 ceph-mon[25688]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:48 smithi164 ceph-mon[24851]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:48.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:48 smithi164 ceph-mon[29512]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:48 smithi172 ceph-mon[25688]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:48.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:50.775 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:50 smithi172 ceph-mon[25688]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:50 smithi164 ceph-mon[24851]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:50.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:50 smithi164 ceph-mon[29512]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:52.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:52 smithi164 ceph-mon[24851]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:52 smithi164 ceph-mon[29512]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:52 smithi172 ceph-mon[25688]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:53.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:53 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:54 smithi164 ceph-mon[24851]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:54 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:32:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:54 smithi164 ceph-mon[29512]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:54 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:32:54.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:54 smithi172 ceph-mon[25688]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:54.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:54 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:32:55.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:55 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:55.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:55 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:55.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:55 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:32:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:32:55] "GET /metrics HTTP/1.1" 200 37976 "" "Prometheus/2.33.4" 2024-03-26T02:32:56.713 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:56 smithi164 ceph-mon[24851]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:56.713 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:56 smithi164 ceph-mon[29512]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:56.782 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:56 smithi172 ceph-mon[25688]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[24851]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:58.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[29512]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:58.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:58.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:58 smithi172 ceph-mon[25688]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:58.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:58 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:58.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:59.048 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:32:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:32:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:32:59.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:59 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:32:59.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:59 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:32:59.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:59 smithi172 ceph-mon[25688]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s 2024-03-26T02:32:59.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:59 smithi172 ceph-mon[25688]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T02:32:59.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:32:59 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:59.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:32:59.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:32:59.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[29512]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s 2024-03-26T02:32:59.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[29512]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T02:32:59.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:32:59.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:32:59.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:32:59.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[24851]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 960 B/s rd, 0 op/s 2024-03-26T02:32:59.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[24851]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T02:32:59.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:32:59 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:33:01.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:01 smithi164 ceph-mon[24851]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:01.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:01 smithi164 ceph-mon[29512]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:01.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:01 smithi172 ceph-mon[25688]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:03.505 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:03 smithi172 ceph-mon[25688]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T02:33:03.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:03 smithi164 ceph-mon[24851]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T02:33:03.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:03 smithi164 ceph-mon[29512]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s 2024-03-26T02:33:03.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:05.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:05 smithi164 ceph-mon[24851]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:05.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:05 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:05.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:05 smithi164 ceph-mon[29512]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:05.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:05 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:05.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:05 smithi172 ceph-mon[25688]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:05 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:05.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:33:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:33:05] "GET /metrics HTTP/1.1" 200 37975 "" "Prometheus/2.33.4" 2024-03-26T02:33:07.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:07 smithi164 ceph-mon[24851]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:07.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:07 smithi164 ceph-mon[29512]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:07.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:07 smithi172 ceph-mon[25688]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:09.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:08 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:09.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:09 smithi164 ceph-mon[24851]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:33:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:09 smithi164 ceph-mon[29512]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:33:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:09 smithi172 ceph-mon[25688]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:33:11.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:11 smithi164 ceph-mon[24851]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:11.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:11 smithi164 ceph-mon[29512]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:11.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:11 smithi172 ceph-mon[25688]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:13.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:13 smithi164 ceph-mon[24851]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:13.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:13 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:13.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:13 smithi164 ceph-mon[29512]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:13.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:13 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:13.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:13 smithi172 ceph-mon[25688]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:13.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:13 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:13.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:13.503Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:15.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:15 smithi164 ceph-mon[24851]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:15.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:15 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:15.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:15 smithi164 ceph-mon[29512]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:15.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:15 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:15.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:15 smithi172 ceph-mon[25688]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:15 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:15.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:33:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:33:15] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.33.4" 2024-03-26T02:33:17.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:17 smithi164 ceph-mon[24851]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:17.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:17 smithi164 ceph-mon[29512]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:17 smithi172 ceph-mon[25688]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:18.955 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:19.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:19 smithi164 ceph-mon[29512]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:19 smithi164 ceph-mon[24851]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:19 smithi172 ceph-mon[25688]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:21 smithi164 ceph-mon[24851]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:21.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:21 smithi164 ceph-mon[29512]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:21 smithi172 ceph-mon[25688]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:23.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:23 smithi164 ceph-mon[24851]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:23.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:23 smithi164 ceph-mon[29512]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:23.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:23 smithi172 ceph-mon[25688]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:23.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:25.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:25 smithi164 ceph-mon[24851]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:25.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:25 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:25 smithi164 ceph-mon[29512]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:25 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:25 smithi172 ceph-mon[25688]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:25.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:25 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:25.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:33:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:33:25] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.33.4" 2024-03-26T02:33:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:27 smithi164 ceph-mon[24851]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:27.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:27 smithi164 ceph-mon[29512]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:27.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:27 smithi172 ceph-mon[25688]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:28 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:28 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:28 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:28.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:28.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:29 smithi164 ceph-mon[24851]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:29 smithi164 ceph-mon[29512]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:29 smithi172 ceph-mon[25688]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:31.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:31 smithi164 ceph-mon[24851]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:31.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:31 smithi164 ceph-mon[29512]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:31.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:31 smithi172 ceph-mon[25688]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:33.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:33 smithi164 ceph-mon[24851]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:33.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:33 smithi164 ceph-mon[29512]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:33.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:33 smithi172 ceph-mon[25688]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:33.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:35.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:35 smithi164 ceph-mon[24851]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:35.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:35 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:35.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:35 smithi164 ceph-mon[29512]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:35.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:35 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:35.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:35 smithi172 ceph-mon[25688]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:35.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:35 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:35.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:33:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:33:35] "GET /metrics HTTP/1.1" 200 37985 "" "Prometheus/2.33.4" 2024-03-26T02:33:37.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:37 smithi164 ceph-mon[24851]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:37.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:37 smithi164 ceph-mon[29512]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:37.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:37 smithi172 ceph-mon[25688]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:39.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:39 smithi164 ceph-mon[24851]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:39 smithi164 ceph-mon[29512]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:39.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:39 smithi172 ceph-mon[25688]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:41.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:41 smithi164 ceph-mon[24851]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:41.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:41 smithi164 ceph-mon[29512]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:41.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:41 smithi172 ceph-mon[25688]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:43.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:43 smithi164 ceph-mon[24851]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:43.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:43 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:43.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:43 smithi164 ceph-mon[29512]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:43.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:43 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:43 smithi172 ceph-mon[25688]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:43 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:43.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:45 smithi164 ceph-mon[24851]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:45 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:45.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:45 smithi164 ceph-mon[29512]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:45.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:45 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:45 smithi172 ceph-mon[25688]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:45 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:45.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:33:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:33:45] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-26T02:33:47.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:47 smithi164 ceph-mon[24851]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:47.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:47 smithi164 ceph-mon[29512]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:47.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:47 smithi172 ceph-mon[25688]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:49.056 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:49 smithi164 ceph-mon[24851]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:49.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:49 smithi164 ceph-mon[29512]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:49 smithi172 ceph-mon[25688]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:51.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:51 smithi164 ceph-mon[24851]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:51 smithi164 ceph-mon[29512]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:51 smithi172 ceph-mon[25688]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:53.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:53 smithi164 ceph-mon[29512]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:53.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:53 smithi164 ceph-mon[24851]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:53.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:53 smithi172 ceph-mon[25688]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:53.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:53 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:55.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:55 smithi164 ceph-mon[24851]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:55.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:55 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:55.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:55 smithi164 ceph-mon[29512]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:55.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:55 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:55.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:55 smithi172 ceph-mon[25688]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:55.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:55 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:33:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:33:55] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-26T02:33:57.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:57 smithi164 ceph-mon[29512]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:57.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:57 smithi164 ceph-mon[24851]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:57.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:57 smithi172 ceph-mon[25688]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:33:58.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:33:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:33:58.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:33:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:33:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:33:59.749 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:33:59 smithi164 ceph-mon[24851]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:59.749 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:33:59 smithi164 ceph-mon[29512]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:33:59 smithi172 ceph-mon[25688]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:00.465 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch daemon redeploy "mgr.$(ceph mgr dump -f json | jq .standbys | jq .[] | jq -r .name)" --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-26T02:34:01.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:01 smithi164 ceph-mon[29512]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:01.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:01 smithi164 ceph-mon[24851]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:01.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:01 smithi172 ceph-mon[25688]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:03.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:03 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:03.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:03 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:03.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:03 smithi164 ceph-mon[24851]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:03.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:03 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:03 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:03 smithi164 ceph-mon[29512]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:03 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:03 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:03 smithi172 ceph-mon[25688]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:03.381 INFO:teuthology.orchestra.run.smithi164.stdout:Scheduled to redeploy mgr.y on host 'smithi164' 2024-03-26T02:34:03.806 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:04.042 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps --refresh' 2024-03-26T02:34:04.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/601192877' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:34:04.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[29512]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:34:04.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[29512]: Schedule redeploy daemon mgr.y 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/601192877' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[24851]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[24851]: Schedule redeploy daemon mgr.y 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:04.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:04 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:04.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:04 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/601192877' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:34:04.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:04 smithi172 ceph-mon[25688]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch daemon redeploy", "name": "mgr.y", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:34:04.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:04 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:04.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:04 smithi172 ceph-mon[25688]: Schedule redeploy daemon mgr.y 2024-03-26T02:34:04.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:04 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:04.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:04 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:05.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:34:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:34:05] "GET /metrics HTTP/1.1" 200 37982 "" "Prometheus/2.33.4" 2024-03-26T02:34:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[24851]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[29512]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:05 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:05 smithi172 ceph-mon[25688]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:05 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:05 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:05 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:05 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:05 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:06.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:05 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:06.758 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:34:06.758 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (2m) 1s ago 10m 25.2M - 0.23.0 ba2b418f427c 98dbe984702c 2024-03-26T02:34:06.758 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 error 4s ago 10m - - 2024-03-26T02:34:06.758 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (2m) 1s ago 9m 55.1M - 3.5 e1d6a67b021e 87294cc93aa4 2024-03-26T02:34:06.758 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283 running (5m) 4s ago 15m 511M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 d86222d71593 2024-03-26T02:34:06.758 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:9283 running (16m) 1s ago 16m 411M - 17.2.0 e1d6a67b021e bdfa3c50c167 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (16m) 1s ago 16m 57.4M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (15m) 4s ago 15m 47.0M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (15m) 1s ago 15m 44.5M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (2m) 1s ago 11m 16.8M - 1.3.1 1dbe0e931976 547c321e8f73 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (2m) 4s ago 11m 17.0M - 1.3.1 1dbe0e931976 114ffb73b278 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (14m) 1s ago 14m 52.2M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (14m) 1s ago 14m 53.1M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (14m) 1s ago 14m 51.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (13m) 1s ago 13m 54.0M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (13m) 4s ago 13m 52.2M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (12m) 4s ago 12m 49.9M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (12m) 4s ago 12m 51.5M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (12m) 4s ago 12m 51.5M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:34:06.759 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (2m) 4s ago 11m 71.5M - 2.33.4 514e6a882f6e 7584c8ee6d15 2024-03-26T02:34:06.760 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (9m) 1s ago 9m 73.3M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:34:06.760 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (9m) 4s ago 9m 75.6M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:34:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[29512]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-26T02:34:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[29512]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:34:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:34:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:34:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:07.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[29512]: Deploying daemon mgr.y on smithi164 2024-03-26T02:34:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[24851]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-26T02:34:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[24851]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:34:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:34:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:34:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:06 smithi164 ceph-mon[24851]: Deploying daemon mgr.y on smithi164 2024-03-26T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:06 smithi172 ceph-mon[25688]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 0 op/s 2024-03-26T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:06 smithi172 ceph-mon[25688]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:06 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:06 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:06 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:06 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:07.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:06 smithi172 ceph-mon[25688]: Deploying daemon mgr.y on smithi164 2024-03-26T02:34:07.429 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'sleep 180' 2024-03-26T02:34:08.099 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:07 smithi164 ceph-mon[24851]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:34:08.100 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:07 smithi164 ceph-mon[29512]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:34:08.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:07 smithi172 ceph-mon[25688]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:34:08.947 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:08 smithi164 ceph-mon[24851]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:08.947 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:08 smithi164 ceph-mon[29512]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:08 smithi172 ceph-mon[25688]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:09.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:08 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:10.289 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:09 smithi164 systemd[1]: Stopping Ceph mgr.y for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:34:10.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:10 smithi164 podman[62167]: 2024-03-26 02:34:10.288026353 +0000 UTC m=+0.181649742 container died bdfa3c50c1674eb7adb393a7a32553f75f820ab40b0684e6995def0d26114bc7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, name=centos-stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.expose-services=, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0) 2024-03-26T02:34:10.926 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:10 smithi164 ceph-mon[24851]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:10.926 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:10 smithi164 ceph-mon[29512]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:10.927 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:10 smithi164 podman[62167]: 2024-03-26 02:34:10.674404722 +0000 UTC m=+0.568028104 container cleanup bdfa3c50c1674eb7adb393a7a32553f75f820ab40b0684e6995def0d26114bc7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, version=8, ceph=True, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, name=centos-stream, distribution-scope=public, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc.) 2024-03-26T02:34:10.927 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:10 smithi164 bash[62167]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y 2024-03-26T02:34:11.286 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:10 smithi164 podman[62182]: 2024-03-26 02:34:10.926164807 +0000 UTC m=+0.638543853 container remove bdfa3c50c1674eb7adb393a7a32553f75f820ab40b0684e6995def0d26114bc7 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, version=8, vcs-type=git, vendor=Red Hat, Inc., architecture=x86_64, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-26T02:34:11.287 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:10 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:34:11.287 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-26T02:34:11.287 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 systemd[1]: Stopped Ceph mgr.y for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:34:11.287 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service: Consumed 58.373s CPU time. 2024-03-26T02:34:11.287 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 systemd[1]: Starting Ceph mgr.y for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:34:11.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:10 smithi172 ceph-mon[25688]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:11.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 podman[62311]: 2024-03-26 02:34:11.404703007 +0000 UTC m=+0.018985659 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:34:11.545 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 podman[62311]: 2024-03-26 02:34:11.519393873 +0000 UTC m=+0.133676554 container create 09bd85f609f26538ecc629d423f3b62b06d695d3cb243535ce7fa58360328aa3 (image=quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T02:34:12.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 podman[62311]: 2024-03-26 02:34:11.755109783 +0000 UTC m=+0.369392426 container init 09bd85f609f26538ecc629d423f3b62b06d695d3cb243535ce7fa58360328aa3 (image=quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T02:34:12.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 podman[62311]: 2024-03-26 02:34:11.757849604 +0000 UTC m=+0.372132251 container start 09bd85f609f26538ecc629d423f3b62b06d695d3cb243535ce7fa58360328aa3 (image=quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T02:34:12.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 ceph-mgr[62330]: -- 172.21.15.164:0/2312423004 <== mon.0 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a1c1a905a0 con 0x55a1c1a64800 2024-03-26T02:34:12.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 bash[62311]: 09bd85f609f26538ecc629d423f3b62b06d695d3cb243535ce7fa58360328aa3 2024-03-26T02:34:12.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:11 smithi164 systemd[1]: Started Ceph mgr.y for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:34:12.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:12.329+0000 7ff72601c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:34:12.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:12.435+0000 7ff72601c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[24851]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[29512]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.157 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.158 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.158 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.158 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:34:13.158 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:12 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:12 smithi172 ceph-mon[25688]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 82 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:34:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:12 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:12 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:12 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:12 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:12 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:34:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:12 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:13.437 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:13.156+0000 7ff72601c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:34:13.437 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:13.253+0000 7ff72601c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:34:13.437 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:13.398+0000 7ff72601c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:34:13.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:13.503+0000 7ff72601c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:34:13.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:13.587+0000 7ff72601c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:34:13.806 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:14.177 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:13.861+0000 7ff72601c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:34:14.178 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:13.973+0000 7ff72601c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:34:14.178 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:14.061+0000 7ff72601c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:34:14.178 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:14.177+0000 7ff72601c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:34:14.540 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:14.366+0000 7ff72601c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:34:14.540 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:14 smithi164 ceph-mon[24851]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-03-26T02:34:14.540 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:14 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:14.540 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:14 smithi164 ceph-mon[29512]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-03-26T02:34:14.541 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:14 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:14.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:14 smithi172 ceph-mon[25688]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 149 op/s 2024-03-26T02:34:14.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:14 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:15.715 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:15 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:15.715 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:34:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:34:15] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-03-26T02:34:15.731 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:15 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:15.731 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:15.393+0000 7ff72601c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:34:15.731 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:15.542+0000 7ff72601c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:34:15.732 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:15 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:15.984 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:15.891+0000 7ff72601c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:34:16.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:15.983+0000 7ff72601c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:34:16.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:16.188+0000 7ff72601c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:34:16.666 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:16.412+0000 7ff72601c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:34:16.667 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:16 smithi164 ceph-mon[24851]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-26T02:34:16.667 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:16 smithi164 ceph-mon[29512]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-26T02:34:16.720 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:16 smithi172 ceph-mon[25688]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s rd, 0 B/s wr, 143 op/s 2024-03-26T02:34:16.934 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:16.718+0000 7ff72601c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:34:17.244 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:16 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:16.985+0000 7ff72601c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:34:17.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:17.243+0000 7ff72601c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:34:18.360 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:18.095+0000 7ff72601c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[24851]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:18.718+0000 7ff72601c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:18.819+0000 7ff72601c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:19.043+0000 7ff72601c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[29512]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:18 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-mon[25688]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:34:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:18 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:34:19.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:34:19] ENGINE Bus STARTING 2024-03-26T02:34:19.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: CherryPy Checker: 2024-03-26T02:34:19.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: The Application mounted at '' has an empty config. 2024-03-26T02:34:19.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:34:19.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:34:19] ENGINE Serving on http://:::9283 2024-03-26T02:34:19.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:34:19] ENGINE Bus STARTED 2024-03-26T02:34:20.232 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:19 smithi172 ceph-mon[25688]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-03-26T02:34:20.232 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:19 smithi172 ceph-mon[25688]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-26T02:34:20.232 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:19 smithi172 ceph-mon[25688]: Standby manager daemon y restarted 2024-03-26T02:34:20.232 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:19 smithi172 ceph-mon[25688]: Standby manager daemon y started 2024-03-26T02:34:20.232 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:19 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:34:20.232 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:19 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:34:20.232 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:19 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:34:20.232 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:19 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:34:20.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[24851]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-03-26T02:34:20.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[24851]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-26T02:34:20.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[24851]: Standby manager daemon y restarted 2024-03-26T02:34:20.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[24851]: Standby manager daemon y started 2024-03-26T02:34:20.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:34:20.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:34:20.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[29512]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 0 B/s wr, 132 op/s 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[29512]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[29512]: Standby manager daemon y restarted 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[29512]: Standby manager daemon y started 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:34:20.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:19 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:34:21.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:20 smithi164 ceph-mon[24851]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-26T02:34:21.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:20 smithi164 ceph-mon[29512]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-26T02:34:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:20 smithi172 ceph-mon[25688]: mgrmap e26: x(active, since 3m), standbys: y 2024-03-26T02:34:22.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:21 smithi164 ceph-mon[24851]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-26T02:34:22.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:21 smithi164 ceph-mon[29512]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-26T02:34:22.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:21 smithi172 ceph-mon[25688]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 161 op/s 2024-03-26T02:34:23.806 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:24.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:23 smithi164 ceph-mon[24851]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-26T02:34:24.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:23 smithi164 ceph-mon[29512]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-26T02:34:24.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:23 smithi172 ceph-mon[25688]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 574 B/s rd, 0 op/s 2024-03-26T02:34:25.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:34:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:34:25] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-03-26T02:34:26.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:25 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:26.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:25 smithi164 ceph-mon[24851]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:34:26.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:25 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:26.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:25 smithi164 ceph-mon[29512]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:34:26.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:25 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:26.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:25 smithi172 ceph-mon[25688]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:34:28.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:27 smithi164 ceph-mon[24851]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:34:28.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:27 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:28.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:27 smithi164 ceph-mon[29512]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:34:28.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:27 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:28.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:27 smithi172 ceph-mon[25688]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:34:28.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:27 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:29.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:29.725 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:29 smithi172 ceph-mon[25688]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:34:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:29 smithi164 ceph-mon[24851]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:34:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:29 smithi164 ceph-mon[29512]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:34:32.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:31 smithi164 ceph-mon[24851]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:32.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:31 smithi164 ceph-mon[29512]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:32.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:31 smithi172 ceph-mon[25688]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:33.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:33 smithi172 ceph-mon[25688]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:33.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:33 smithi164 ceph-mon[24851]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:34.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:33 smithi164 ceph-mon[29512]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:35.724 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:34:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:34:35] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-26T02:34:36.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:35 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:36.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:35 smithi164 ceph-mon[29512]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:35 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:35 smithi164 ceph-mon[24851]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:35 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:35 smithi172 ceph-mon[25688]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:37 smithi164 ceph-mon[24851]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:38.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:37 smithi164 ceph-mon[29512]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:37 smithi172 ceph-mon[25688]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:39.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:39 smithi164 ceph-mon[24851]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:39 smithi164 ceph-mon[29512]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:39.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:39 smithi172 ceph-mon[25688]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:42.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:41 smithi164 ceph-mon[24851]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:42.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:41 smithi164 ceph-mon[29512]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:42.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:41 smithi172 ceph-mon[25688]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:42 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:42 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:42 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:43.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:43 smithi172 ceph-mon[25688]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:43.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:43 smithi164 ceph-mon[24851]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:44.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:43 smithi164 ceph-mon[29512]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:45.732 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:34:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:34:45] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-26T02:34:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:45 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:45 smithi164 ceph-mon[29512]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:45 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:45 smithi164 ceph-mon[24851]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:45 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:45 smithi172 ceph-mon[25688]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:48.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:47 smithi164 ceph-mon[24851]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:48.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:47 smithi164 ceph-mon[29512]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:48.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:47 smithi172 ceph-mon[25688]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:49.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:49 smithi164 ceph-mon[24851]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:49.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:49 smithi164 ceph-mon[29512]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:49 smithi172 ceph-mon[25688]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:52.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:51 smithi164 ceph-mon[24851]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:52.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:51 smithi164 ceph-mon[29512]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:52.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:51 smithi172 ceph-mon[25688]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:53.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:53 smithi172 ceph-mon[25688]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:53.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:53 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:54.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:53 smithi164 ceph-mon[24851]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:53 smithi164 ceph-mon[29512]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:55.732 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:34:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:34:55] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-26T02:34:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:55 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:55 smithi164 ceph-mon[29512]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:56.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:55 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:56.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:55 smithi164 ceph-mon[24851]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:55 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:55 smithi172 ceph-mon[25688]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:57 smithi164 ceph-mon[24851]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:57 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:58.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:57 smithi164 ceph-mon[29512]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:58.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:57 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:58.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:57 smithi172 ceph-mon[25688]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:34:58.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:57 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:59.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:34:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:34:58.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:34:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:34:59 smithi164 ceph-mon[24851]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:59.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:34:59 smithi164 ceph-mon[29512]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:34:59 smithi172 ceph-mon[25688]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:02.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:01 smithi164 ceph-mon[24851]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:02.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:01 smithi164 ceph-mon[29512]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:02.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:01 smithi172 ceph-mon[25688]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:03.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:03 smithi172 ceph-mon[25688]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:03.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:04.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:03 smithi164 ceph-mon[29512]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:04.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:03 smithi164 ceph-mon[24851]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:05.735 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:35:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:35:05] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-26T02:35:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:05 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:05 smithi164 ceph-mon[24851]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:05 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:05 smithi164 ceph-mon[29512]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:05 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:05 smithi172 ceph-mon[25688]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:08.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:07 smithi164 ceph-mon[24851]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:08.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:07 smithi164 ceph-mon[29512]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:07 smithi172 ceph-mon[25688]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:09.056 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:08 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:09.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:09 smithi164 ceph-mon[24851]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:09 smithi164 ceph-mon[29512]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:09 smithi172 ceph-mon[25688]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:12.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:11 smithi164 ceph-mon[24851]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:12.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:11 smithi164 ceph-mon[29512]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:12.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:11 smithi172 ceph-mon[25688]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:12 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:13.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:12 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:12 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:13.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:13 smithi172 ceph-mon[25688]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:13.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:13 smithi164 ceph-mon[24851]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:14.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:13 smithi164 ceph-mon[29512]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:15.740 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:35:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:35:15] "GET /metrics HTTP/1.1" 200 37918 "" "Prometheus/2.33.4" 2024-03-26T02:35:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:15 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:15 smithi164 ceph-mon[24851]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:16.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:15 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:15 smithi164 ceph-mon[29512]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:15 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:15 smithi172 ceph-mon[25688]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:18.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:17 smithi164 ceph-mon[24851]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:17 smithi164 ceph-mon[29512]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:17 smithi172 ceph-mon[25688]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:19.056 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:19.667 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:19 smithi164 ceph-mon[24851]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:19.667 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:19 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:35:19.668 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:19 smithi164 ceph-mon[29512]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:19.668 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:19 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:35:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:19 smithi172 ceph-mon[25688]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:19 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:35:22.017 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:21 smithi164 ceph-mon[24851]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:22.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:21 smithi164 ceph-mon[29512]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:22.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:21 smithi172 ceph-mon[25688]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:23 smithi172 ceph-mon[25688]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:23 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:23 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:23 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:35:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:23 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:35:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:23 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:23.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[24851]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[29512]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:35:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:23 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:35:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:24 smithi164 ceph-mon[24851]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:35:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:24 smithi164 ceph-mon[24851]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:24 smithi164 ceph-mon[29512]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:35:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:24 smithi164 ceph-mon[29512]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:25.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:24 smithi172 ceph-mon[25688]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:35:25.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:24 smithi172 ceph-mon[25688]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:25.742 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:35:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:35:25] "GET /metrics HTTP/1.1" 200 37918 "" "Prometheus/2.33.4" 2024-03-26T02:35:26.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:25 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:25 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:25 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:27.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:26 smithi164 ceph-mon[29512]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:26 smithi164 ceph-mon[24851]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:27.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:26 smithi172 ceph-mon[25688]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:27 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:27 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:28.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:27 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:28 smithi164 ceph-mon[24851]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-03-26T02:35:29.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:28 smithi164 ceph-mon[29512]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-03-26T02:35:29.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:28 smithi172 ceph-mon[25688]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-03-26T02:35:29.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:30 smithi164 ceph-mon[29512]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:30 smithi164 ceph-mon[24851]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:30 smithi172 ceph-mon[25688]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:33.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:32 smithi164 ceph-mon[29512]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:32 smithi164 ceph-mon[24851]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:33.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:32 smithi172 ceph-mon[25688]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:35:33.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:34 smithi164 ceph-mon[24851]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:34 smithi164 ceph-mon[29512]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:34 smithi172 ceph-mon[25688]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:35.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:35 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:35.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:35 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:35.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:35 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:35.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:35:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:35:35] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-26T02:35:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:36 smithi164 ceph-mon[24851]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:36 smithi164 ceph-mon[29512]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:36 smithi172 ceph-mon[25688]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:38 smithi164 ceph-mon[24851]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:38 smithi164 ceph-mon[29512]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:38 smithi172 ceph-mon[25688]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:39.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:40 smithi164 ceph-mon[24851]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:41.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:40 smithi164 ceph-mon[29512]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:40 smithi172 ceph-mon[25688]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:42 smithi164 ceph-mon[29512]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:42 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:42 smithi164 ceph-mon[24851]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:42 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:42 smithi172 ceph-mon[25688]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:42 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:43.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:44 smithi164 ceph-mon[24851]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:44.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:44 smithi164 ceph-mon[29512]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:44 smithi172 ceph-mon[25688]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:45 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:45.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:45 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:45.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:45 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:45.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:35:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:35:45] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-26T02:35:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:46 smithi164 ceph-mon[24851]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:46 smithi164 ceph-mon[29512]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:46 smithi172 ceph-mon[25688]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:48 smithi164 ceph-mon[24851]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:48 smithi164 ceph-mon[29512]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:48 smithi172 ceph-mon[25688]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:49.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:50 smithi164 ceph-mon[24851]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:51.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:50 smithi164 ceph-mon[29512]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:50 smithi172 ceph-mon[25688]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:52 smithi164 ceph-mon[24851]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:53.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:52 smithi164 ceph-mon[29512]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:52 smithi172 ceph-mon[25688]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:53.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:53 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:35:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:54 smithi164 ceph-mon[24851]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:54 smithi164 ceph-mon[29512]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:54.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:54 smithi172 ceph-mon[25688]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:55.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:55 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:55.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:55 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:55.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:55 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:35:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:35:55] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-26T02:35:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:56 smithi164 ceph-mon[24851]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:56.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:56 smithi164 ceph-mon[29512]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:56.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:56 smithi172 ceph-mon[25688]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:57 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:58.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:57 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:58.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:57 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:35:58 smithi164 ceph-mon[24851]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:59.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:35:58 smithi164 ceph-mon[29512]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:35:58 smithi172 ceph-mon[25688]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:59.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:35:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:35:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:36:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:00 smithi164 ceph-mon[24851]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:01.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:00 smithi164 ceph-mon[29512]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:01.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:00 smithi172 ceph-mon[25688]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:03.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:02 smithi164 ceph-mon[24851]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:03.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:02 smithi164 ceph-mon[29512]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:03.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:02 smithi172 ceph-mon[25688]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:03.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:36:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:36:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:36:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:04 smithi164 ceph-mon[24851]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:04 smithi164 ceph-mon[29512]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:04 smithi172 ceph-mon[25688]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:05.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:05 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:05.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:05 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:05.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:05 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:05.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:36:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:36:05] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-26T02:36:06.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:06 smithi164 ceph-mon[24851]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:06.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:06 smithi164 ceph-mon[29512]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:06 smithi172 ceph-mon[25688]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:08 smithi164 ceph-mon[24851]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:09.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:08 smithi164 ceph-mon[29512]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:08 smithi172 ceph-mon[25688]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:09.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:36:08 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:36:08.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:36:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:10 smithi164 ceph-mon[24851]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:11.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:10 smithi164 ceph-mon[29512]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:10 smithi172 ceph-mon[25688]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:12 smithi164 ceph-mon[24851]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:12 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:13.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:12 smithi164 ceph-mon[29512]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:13.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:12 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:12 smithi172 ceph-mon[25688]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:12 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:13.806 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:36:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:36:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:36:14.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:14 smithi164 ceph-mon[24851]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:14.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:14 smithi164 ceph-mon[29512]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:14.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:14 smithi172 ceph-mon[25688]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:15.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:15 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:15.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:15 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:15.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:15 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:15.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:36:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:36:15] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-26T02:36:16.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:16 smithi164 ceph-mon[24851]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:16.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:16 smithi164 ceph-mon[29512]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:16 smithi172 ceph-mon[25688]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:18 smithi164 ceph-mon[24851]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:18 smithi164 ceph-mon[29512]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:18 smithi172 ceph-mon[25688]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:19.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:36:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:36:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"172.21.15.172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", cluster_addr=\"172.21.15.164\", device_class=\"ssd\", hostname=\"smithi164\", instance=\"smithi172:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.164\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:36:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:20 smithi164 ceph-mon[24851]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:20 smithi164 ceph-mon[29512]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:20 smithi172 ceph-mon[25688]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:22 smithi164 ceph-mon[24851]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:22 smithi164 ceph-mon[29512]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:23.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:22 smithi172 ceph-mon[25688]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:23.806 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:36:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:36:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi164\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"e2654d3c-eb16-11ee-95d0-87774f69a715\", domainname=\"(none)\", instance=\"smithi164\", job=\"node\", machine=\"x86_64\", nodename=\"smithi164\", release=\"5.14.0-432.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-03-26T02:36:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[24851]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:36:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:36:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:36:24.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[24851]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:36:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[29512]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:36:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:36:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:36:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:24 smithi164 ceph-mon[29512]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:36:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:24 smithi172 ceph-mon[25688]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:24 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:36:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:24 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:36:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:24 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:36:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:24 smithi172 ceph-mon[25688]: from='mgr.34751 ' entity='mgr.x' 2024-03-26T02:36:25.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:25 smithi164 ceph-mon[24851]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-26T02:36:25.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:25 smithi164 ceph-mon[24851]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:25.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:25 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:25 smithi164 ceph-mon[29512]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-26T02:36:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:25 smithi164 ceph-mon[29512]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:25.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:25 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:25 smithi172 ceph-mon[25688]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 952 B/s rd, 0 op/s 2024-03-26T02:36:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:25 smithi172 ceph-mon[25688]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:25.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:25 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:25.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:36:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:36:25] "GET /metrics HTTP/1.1" 200 37987 "" "Prometheus/2.33.4" 2024-03-26T02:36:27.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:27 smithi164 ceph-mon[24851]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:27.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:27 smithi164 ceph-mon[29512]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:27.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:27 smithi172 ceph-mon[25688]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:28 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:28 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:28 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:29 smithi164 ceph-mon[24851]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-26T02:36:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:29 smithi164 ceph-mon[29512]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-26T02:36:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:29 smithi172 ceph-mon[25688]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-26T02:36:31.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:31 smithi164 ceph-mon[24851]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:31.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:31 smithi164 ceph-mon[29512]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:31.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:31 smithi172 ceph-mon[25688]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:36:33.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:33 smithi164 ceph-mon[24851]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-26T02:36:33.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:33 smithi164 ceph-mon[29512]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-26T02:36:33.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:33 smithi172 ceph-mon[25688]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-03-26T02:36:35.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:35 smithi164 ceph-mon[24851]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:36:35.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:35 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:35.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:35 smithi164 ceph-mon[29512]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:36:35.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:35 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:35.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:35 smithi172 ceph-mon[25688]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:36:35.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:35 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:35.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:36:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:36:35] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-26T02:36:37.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:37 smithi164 ceph-mon[24851]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:37.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:37 smithi164 ceph-mon[29512]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:37.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:37 smithi172 ceph-mon[25688]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:39 smithi164 ceph-mon[24851]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:39 smithi164 ceph-mon[29512]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:39.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:39 smithi172 ceph-mon[25688]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:41.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:41 smithi164 ceph-mon[24851]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:41.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:41 smithi164 ceph-mon[29512]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:41.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:41 smithi172 ceph-mon[25688]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:43.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:43 smithi164 ceph-mon[24851]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:43.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:43 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:43.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:43 smithi164 ceph-mon[29512]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:43.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:43 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:43.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:43 smithi172 ceph-mon[25688]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:43.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:43 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:45 smithi164 ceph-mon[24851]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:45 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:45.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:45 smithi164 ceph-mon[29512]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:45.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:45 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:45.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:45 smithi172 ceph-mon[25688]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:45.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:45 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:45.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:36:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:36:45] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-26T02:36:47.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:47 smithi164 ceph-mon[24851]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:47.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:47 smithi164 ceph-mon[29512]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:47.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:47 smithi172 ceph-mon[25688]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:49 smithi164 ceph-mon[24851]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:49.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:49 smithi164 ceph-mon[29512]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:49 smithi172 ceph-mon[25688]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:51.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:51 smithi164 ceph-mon[24851]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:51 smithi164 ceph-mon[29512]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:51 smithi172 ceph-mon[25688]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:53.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:53 smithi164 ceph-mon[24851]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:53.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:53 smithi164 ceph-mon[29512]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:53.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:53 smithi172 ceph-mon[25688]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:55.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:55 smithi164 ceph-mon[24851]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:55.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:55 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:55.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:55 smithi164 ceph-mon[29512]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:55.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:55 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:55.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:55 smithi172 ceph-mon[25688]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:55.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:55 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:36:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:36:55] "GET /metrics HTTP/1.1" 200 37981 "" "Prometheus/2.33.4" 2024-03-26T02:36:57.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:57 smithi164 ceph-mon[24851]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:57.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:57 smithi164 ceph-mon[29512]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:57.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:57 smithi172 ceph-mon[25688]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:58 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:58.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:58 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:58 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:36:59 smithi164 ceph-mon[24851]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:59.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:36:59 smithi164 ceph-mon[29512]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:36:59 smithi172 ceph-mon[25688]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:01.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:01 smithi164 ceph-mon[24851]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:01.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:01 smithi164 ceph-mon[29512]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:01.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:01 smithi172 ceph-mon[25688]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:03.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:03 smithi164 ceph-mon[24851]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:03.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:03 smithi164 ceph-mon[29512]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:03.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:03 smithi172 ceph-mon[25688]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:05.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:05 smithi164 ceph-mon[24851]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:05.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:05 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:05.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:05 smithi164 ceph-mon[29512]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:05.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:05 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:05.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:05 smithi172 ceph-mon[25688]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:05.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:05 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:05.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:37:05] "GET /metrics HTTP/1.1" 200 37984 "" "Prometheus/2.33.4" 2024-03-26T02:37:07.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:07 smithi164 ceph-mon[24851]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:07.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:07 smithi164 ceph-mon[29512]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:07.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:07 smithi172 ceph-mon[25688]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:09.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:09 smithi164 ceph-mon[24851]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:09.813 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:09 smithi164 ceph-mon[29512]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:09.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:09 smithi172 ceph-mon[25688]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:10.533 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:37:11.781 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:11 smithi164 ceph-mon[24851]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:11.782 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:11 smithi164 ceph-mon[29512]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:11.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:11 smithi172 ceph-mon[25688]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:12.494 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:37:12.494 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (5m) 109s ago 13m 25.2M - 0.23.0 ba2b418f427c 98dbe984702c 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 error 2m ago 13m - - 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (5m) 109s ago 12m 55.5M - 3.5 e1d6a67b021e 87294cc93aa4 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283 running (8m) 2m ago 18m 512M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 d86222d71593 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (3m) 109s ago 19m 451M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (19m) 109s ago 19m 59.5M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (18m) 2m ago 18m 47.9M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (18m) 109s ago 18m 40.1M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (6m) 109s ago 14m 18.6M - 1.3.1 1dbe0e931976 547c321e8f73 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (5m) 2m ago 14m 17.0M - 1.3.1 1dbe0e931976 114ffb73b278 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (17m) 109s ago 17m 53.0M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (17m) 109s ago 17m 54.2M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (17m) 109s ago 17m 52.0M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:37:12.495 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (16m) 109s ago 16m 55.1M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:37:12.496 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (16m) 2m ago 16m 52.9M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:37:12.496 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (16m) 2m ago 16m 50.7M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:37:12.496 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (15m) 2m ago 15m 51.4M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:37:12.496 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (15m) 2m ago 15m 52.6M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:37:12.496 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (5m) 2m ago 14m 71.5M - 2.33.4 514e6a882f6e 7584c8ee6d15 2024-03-26T02:37:12.496 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (13m) 109s ago 13m 73.7M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:37:12.496 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (13m) 2m ago 13m 75.7M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:37:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:12 smithi164 ceph-mon[24851]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:12.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:12 smithi164 ceph-mon[29512]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:12.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:12 smithi172 ceph-mon[25688]: from='mgr.34751 172.21.15.172:0/4256650973' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:13.110 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions' 2024-03-26T02:37:13.714 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:13 smithi164 ceph-mon[29512]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:13.714 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:13 smithi164 ceph-mon[29512]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:37:13.714 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:13 smithi164 ceph-mon[24851]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:13.715 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:13 smithi164 ceph-mon[24851]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:37:13.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:13 smithi172 ceph-mon[25688]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:13.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:13 smithi172 ceph-mon[25688]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:37:15.141 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:37:15.141 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:37:15.141 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:37:15.141 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:37:15.141 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:37:15.141 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:37:15.141 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:37:15.142 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:37:15.771 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:15 smithi164 ceph-mon[29512]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:15.771 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:15 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:15.772 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:15 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/610952643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:37:15.772 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:15 smithi164 ceph-mon[24851]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:15.772 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:15 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:15.772 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:15 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/610952643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:37:15.773 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:37:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:15 smithi172 ceph-mon[25688]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:15 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:15 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/610952643' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:37:15.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:37:15] "GET /metrics HTTP/1.1" 200 37983 "" "Prometheus/2.33.4" 2024-03-26T02:37:17.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:17 smithi164 ceph-mon[24851]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:17.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:17 smithi164 ceph-mon[29512]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:17.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:17 smithi172 ceph-mon[25688]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:17.889 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T02:37:17.889 INFO:teuthology.orchestra.run.smithi164.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T02:37:17.889 INFO:teuthology.orchestra.run.smithi164.stdout: daemon grafana.a on smithi172 is in error state 2024-03-26T02:37:18.521 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph -s' 2024-03-26T02:37:18.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:18 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3494214975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:37:18.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:18 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3494214975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:37:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:18 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3494214975' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:37:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:19 smithi164 ceph-mon[24851]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:19.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:19 smithi164 ceph-mon[29512]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:19 smithi172 ceph-mon[25688]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:20.612 INFO:teuthology.orchestra.run.smithi164.stdout: cluster: 2024-03-26T02:37:20.612 INFO:teuthology.orchestra.run.smithi164.stdout: id: e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:37:20.612 INFO:teuthology.orchestra.run.smithi164.stdout: health: HEALTH_WARN 2024-03-26T02:37:20.612 INFO:teuthology.orchestra.run.smithi164.stdout: 1 failed cephadm daemon(s) 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: services: 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: mon: 3 daemons, quorum a,c,b (age 18m) 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: mgr: x(active, since 6m), standbys: y 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: osd: 8 osds: 8 up (since 15m), 8 in (since 15m) 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: data: 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: pools: 6 pools, 161 pgs 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: objects: 241 objects, 585 KiB 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: usage: 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: pgs: 161 active+clean 2024-03-26T02:37:20.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:37:20.614 INFO:teuthology.orchestra.run.smithi164.stdout: io: 2024-03-26T02:37:20.614 INFO:teuthology.orchestra.run.smithi164.stdout: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-26T02:37:20.614 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:37:21.209 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph mgr fail' 2024-03-26T02:37:21.765 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:21 smithi164 ceph-mon[24851]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:21.765 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:21 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2692406837' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:37:21.766 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:21 smithi164 ceph-mon[29512]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:21.766 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:21 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2692406837' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:37:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:21 smithi172 ceph-mon[25688]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:21 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2692406837' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:37:23.792 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-mon[29512]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:23.792 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/54477579' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:37:23.792 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-mon[29512]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T02:37:23.793 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-mon[24851]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:23.793 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/54477579' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:37:23.793 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-mon[24851]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T02:37:23.793 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:37:23] ENGINE Bus STOPPING 2024-03-26T02:37:23.793 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:37:23] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:37:23.793 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:37:23] ENGINE Bus STOPPED 2024-03-26T02:37:23.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:37:23] ENGINE Bus STARTING 2024-03-26T02:37:23.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:37:23] ENGINE Serving on http://:::9283 2024-03-26T02:37:23.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:37:23] ENGINE Bus STARTED 2024-03-26T02:37:23.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:23 smithi172 ceph-mon[25688]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:23 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/54477579' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:37:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:23 smithi172 ceph-mon[25688]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T02:37:23.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:23.548+0000 7f958c4d6700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-26T02:37:23.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ignoring --setuser ceph since I am not root 2024-03-26T02:37:23.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ignoring --setgroup ceph since I am not root 2024-03-26T02:37:23.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:23 smithi172 ceph-mgr[45916]: -- 172.21.15.172:0/1927859614 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56030dece5a0 con 0x56030dea2800 2024-03-26T02:37:24.276 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'sleep 180' 2024-03-26T02:37:24.548 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:24.137+0000 7f968770a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:37:24.548 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:24.235+0000 7f968770a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:37:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/54477579' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:37:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: mgrmap e27: y(active, starting, since 0.328693s) 2024-03-26T02:37:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:37:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:37:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:37:24.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:37:24.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: Manager daemon y is now available 2024-03-26T02:37:24.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:24.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:24.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:37:24.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:37:24.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:37:24.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:37:24.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/54477579' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:37:24.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: mgrmap e27: y(active, starting, since 0.328693s) 2024-03-26T02:37:24.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:37:24.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:37:24.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: Manager daemon y is now available 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:37:24.866 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/54477579' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: mgrmap e27: y(active, starting, since 0.328693s) 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:37:24.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: Manager daemon y is now available 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:37:24.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:37:24.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:37:24.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:24 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:37:25.163 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:24.920+0000 7f968770a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:37:25.163 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:25.015+0000 7f968770a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:37:25.163 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:25.162+0000 7f968770a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:37:25.441 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:25.275+0000 7f968770a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:37:25.441 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:25.360+0000 7f968770a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:37:25.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[24851]: [26/Mar/2024:02:37:24] ENGINE Bus STARTING 2024-03-26T02:37:25.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[24851]: mgrmap e28: y(active, since 1.33556s) 2024-03-26T02:37:25.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[24851]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:25.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[24851]: [26/Mar/2024:02:37:24] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:37:25.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[24851]: [26/Mar/2024:02:37:24] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:37:25.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[24851]: [26/Mar/2024:02:37:24] ENGINE Bus STARTED 2024-03-26T02:37:25.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[24851]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:25.704 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[29512]: [26/Mar/2024:02:37:24] ENGINE Bus STARTING 2024-03-26T02:37:25.705 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[29512]: mgrmap e28: y(active, since 1.33556s) 2024-03-26T02:37:25.705 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[29512]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:25.705 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[29512]: [26/Mar/2024:02:37:24] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:37:25.705 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[29512]: [26/Mar/2024:02:37:24] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:37:25.705 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[29512]: [26/Mar/2024:02:37:24] ENGINE Bus STARTED 2024-03-26T02:37:25.705 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:25 smithi164 ceph-mon[29512]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:25.754 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:25.639+0000 7f968770a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:37:25.754 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:25.753+0000 7f968770a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:37:25.754 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-mon[25688]: [26/Mar/2024:02:37:24] ENGINE Bus STARTING 2024-03-26T02:37:25.754 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-mon[25688]: mgrmap e28: y(active, since 1.33556s) 2024-03-26T02:37:25.754 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-mon[25688]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:25.754 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-mon[25688]: [26/Mar/2024:02:37:24] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:37:25.754 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-mon[25688]: [26/Mar/2024:02:37:24] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:37:25.754 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-mon[25688]: [26/Mar/2024:02:37:24] ENGINE Bus STARTED 2024-03-26T02:37:25.755 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-mon[25688]: from='client.15075 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:26.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:25.840+0000 7f968770a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:37:26.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:25.946+0000 7f968770a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:37:26.501 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:26 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:26.157+0000 7f968770a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:37:26.675 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:26 smithi164 ceph-mon[24851]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:37:26.675 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:26 smithi164 ceph-mon[24851]: Cluster is now healthy 2024-03-26T02:37:26.676 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:26 smithi164 ceph-mon[24851]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:26.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:26 smithi164 ceph-mon[29512]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:37:26.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:26 smithi164 ceph-mon[29512]: Cluster is now healthy 2024-03-26T02:37:26.676 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:26 smithi164 ceph-mon[29512]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:26 smithi172 ceph-mon[25688]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:37:26.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:26 smithi172 ceph-mon[25688]: Cluster is now healthy 2024-03-26T02:37:26.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:26 smithi172 ceph-mon[25688]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:27.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:27.157+0000 7f968770a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:37:27.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:27.290+0000 7f968770a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:37:27.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:27 smithi164 ceph-mon[24851]: mgrmap e29: y(active, since 3s) 2024-03-26T02:37:27.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:27 smithi164 ceph-mon[29512]: mgrmap e29: y(active, since 3s) 2024-03-26T02:37:27.983 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:27 smithi172 ceph-mon[25688]: mgrmap e29: y(active, since 3s) 2024-03-26T02:37:27.983 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:27.666+0000 7f968770a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:37:27.983 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:27.768+0000 7f968770a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:37:27.983 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:27.982+0000 7f968770a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:37:28.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:28.181+0000 7f968770a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:37:28.772 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:28 smithi172 ceph-mon[25688]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:28.772 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:28 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:28.772 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:28 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:28.772 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:28.490+0000 7f968770a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:37:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:28 smithi164 ceph-mon[24851]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:28 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:28 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:28 smithi164 ceph-mon[29512]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:28 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:28 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:29.025 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:28.770+0000 7f968770a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:37:29.025 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:29.023+0000 7f968770a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:37:30.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[24851]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:30.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[29512]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:30.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:29 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:29 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:29 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:29 smithi172 ceph-mon[25688]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:30.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:29 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:29 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:30.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:29.839+0000 7f968770a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:37:30.750 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:30.423+0000 7f968770a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:37:30.750 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:30.526+0000 7f968770a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:37:30.750 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:30.749+0000 7f968770a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: Standby manager daemon x started 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:37:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:37:31.058 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:37:30] ENGINE Bus STARTING 2024-03-26T02:37:31.058 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: CherryPy Checker: 2024-03-26T02:37:31.058 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: The Application mounted at '' has an empty config. 2024-03-26T02:37:31.058 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:37:31.058 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:37:30] ENGINE Serving on http://:::9283 2024-03-26T02:37:31.058 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:37:30] ENGINE Bus STARTED 2024-03-26T02:37:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: Standby manager daemon x started 2024-03-26T02:37:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:37:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: Standby manager daemon x started 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:37:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:37:31.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:37:31.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:31.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:31.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:31.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:31.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:37:31.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:30 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:37:32.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: mgrmap e30: y(active, since 7s), standbys: x 2024-03-26T02:37:32.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:37:32.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:32.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: mgrmap e30: y(active, since 7s), standbys: x 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:37:32.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:32.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:32.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:31 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:32.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: mgrmap e30: y(active, since 7s), standbys: x 2024-03-26T02:37:32.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:32.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:31 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:32 smithi164 ceph-mon[24851]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:37:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:32 smithi164 ceph-mon[24851]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:37:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:32 smithi164 ceph-mon[24851]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:37:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:32 smithi164 ceph-mon[24851]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:37:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:32 smithi164 ceph-mon[29512]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:37:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:32 smithi164 ceph-mon[29512]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:37:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:32 smithi164 ceph-mon[29512]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:37:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:32 smithi164 ceph-mon[29512]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:37:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:32 smithi172 ceph-mon[25688]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:37:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:32 smithi172 ceph-mon[25688]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:37:33.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:32 smithi172 ceph-mon[25688]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:37:33.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:32 smithi172 ceph-mon[25688]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:37:35.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:37:35.024+0000 7ff6abc7a700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Creating ceph-iscsi config... 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Traceback (most recent call last): 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: return _run_code(code, main_globals, None, 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: exec(code, run_globals) 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:37:35.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:37:35.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:37:35.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:37:35.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:37:35.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:37:35.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:37:35.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:35.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Traceback (most recent call last): 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2425, in _daemon_action 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: return self.event_loop.get_result(coro, timeout) 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: return future.result(timeout) 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: return self.__get_result() 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: raise self._exception 2024-03-26T02:37:35.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: config_blobs=daemon_spec.final_config, 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Creating ceph-iscsi config... 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: Traceback (most recent call last): 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: return _run_code(code, main_globals, None, 2024-03-26T02:37:35.298 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: exec(code, run_globals) 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:35.299 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:35.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-mon[24851]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:37:35.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:35 smithi164 ceph-mon[29512]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:37:35.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:35 smithi172 ceph-mon[25688]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:37:35.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:37:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:37:35] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:36 smithi164 ceph-mon[24851]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Creating ceph-iscsi config... 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:37:36.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2425, in _daemon_action 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: return future.result(timeout) 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: return self.__get_result() 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: raise self._exception 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T02:37:36.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: Creating ceph-iscsi config... 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:37:36.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:36 smithi164 ceph-mon[24851]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:37:36.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:36 smithi164 ceph-mon[24851]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:36 smithi164 ceph-mon[29512]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: Creating ceph-iscsi config... 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:37:36.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T02:37:36.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2425, in _daemon_action 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: return future.result(timeout) 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: return self.__get_result() 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: raise self._exception 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-26T02:37:36.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: Creating ceph-iscsi config... 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:37:36.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:36 smithi164 ceph-mon[29512]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:37:36.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:36 smithi164 ceph-mon[29512]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:36 smithi172 ceph-mon[25688]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Creating ceph-iscsi config... 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:37:36.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: exec(code, run_globals) 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2425, in _daemon_action 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: return future.result(timeout) 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: return self.__get_result() 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: raise self._exception 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T02:37:36.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Creating ceph-iscsi config... 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: exec(code, run_globals) 2024-03-26T02:37:36.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:36 smithi172 ceph-mon[25688]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:37:36.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:36 smithi172 ceph-mon[25688]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:37:37.210 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:37.100Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-26T02:37:37.211 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:37.101Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-26T02:37:37.211 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:37.104Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-26T02:37:37.211 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:37.104Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-26T02:37:37.211 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:37.106Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.172:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.172:8765: connect: connection refused" 2024-03-26T02:37:37.512 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:37 smithi172 ceph-mon[25688]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:37.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:37 smithi164 ceph-mon[24851]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:37.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:37 smithi164 ceph-mon[29512]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:38.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 systemd[1]: Stopping Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:37:38.557 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 bash[54878]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[49234]: ts=2024-03-26T02:37:38.503Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T02:37:38.558 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 podman[54891]: 2024-03-26 02:37:38.513234858 +0000 UTC m=+0.110322383 container died 7584c8ee6d15c843a1ca80ac473c064c5259bbd0c2eeadd41b72a9ebe9b94374 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:37:39.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 podman[54891]: 2024-03-26 02:37:38.894882383 +0000 UTC m=+0.491969965 container cleanup 7584c8ee6d15c843a1ca80ac473c064c5259bbd0c2eeadd41b72a9ebe9b94374 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:37:39.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 bash[54891]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:37:39.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:38 smithi172 bash[54918]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:37:39.462 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 podman[54905]: 2024-03-26 02:37:39.162413775 +0000 UTC m=+0.652125277 container remove 7584c8ee6d15c843a1ca80ac473c064c5259bbd0c2eeadd41b72a9ebe9b94374 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:37:39.462 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T02:37:39.462 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 systemd[1]: Stopped Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:37:39.462 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.974s CPU time. 2024-03-26T02:37:39.462 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 systemd[1]: Starting Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:37:39.462 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 podman[54989]: 2024-03-26 02:37:39.29273139 +0000 UTC m=+0.018858306 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T02:37:39.462 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 podman[54989]: 2024-03-26 02:37:39.396577758 +0000 UTC m=+0.122704673 container create cc7530d80e7a2fa5f52519c80711e1ba884f4cdfb5df9a6a3440dca8958130c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:37:39.732 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-mon[25688]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:39.732 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:39.732 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 podman[54989]: 2024-03-26 02:37:39.640454752 +0000 UTC m=+0.366581669 container init cc7530d80e7a2fa5f52519c80711e1ba884f4cdfb5df9a6a3440dca8958130c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:37:39.732 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 podman[54989]: 2024-03-26 02:37:39.643172845 +0000 UTC m=+0.369299770 container start cc7530d80e7a2fa5f52519c80711e1ba884f4cdfb5df9a6a3440dca8958130c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:37:39.732 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.666Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T02:37:39.732 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.666Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.666Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.666Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi172 (none))" 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.666Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.666Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.668Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.669Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.670Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.672Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.672Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.926µs 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.672Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:37:39.733 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 bash[54989]: cc7530d80e7a2fa5f52519c80711e1ba884f4cdfb5df9a6a3440dca8958130c8 2024-03-26T02:37:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:39 smithi164 ceph-mon[24851]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:39 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:39.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:39 smithi164 ceph-mon[29512]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:39.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:39 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:40.056 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 systemd[1]: Started Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:37:40.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:39.861Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-03-26T02:37:40.352 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.098Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-03-26T02:37:40.352 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.326Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-03-26T02:37:40.352 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.327Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-03-26T02:37:40.353 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.327Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=34.261µs wal_replay_duration=654.945686ms total_replay_duration=654.993416ms 2024-03-26T02:37:40.353 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.331Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:37:40.353 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.331Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T02:37:40.353 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.331Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:37:40.353 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.352Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.806549ms db_storage=1.177µs remote_storage=1.278µs web_handler=304ns query_engine=779ns scrape=401.856µs scrape_sd=159.83µs notify=18.654µs notify_sd=16.954µs rules=19.782264ms 2024-03-26T02:37:40.353 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:37:40.352Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T02:37:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:41.079 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:41.079 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:37:41.079 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:37:41.079 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:41.079 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:41.079 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:41.080 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:37:41.080 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:37:41.080 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:40 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:41.080 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:41.080 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:41.080 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:37:41.080 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:37:41.081 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:40 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:42.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:41 smithi164 ceph-mon[24851]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:42.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:41 smithi164 ceph-mon[29512]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:42.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:41 smithi172 ceph-mon[25688]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:37:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 496 B/s rd, 0 op/s 2024-03-26T02:37:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 496 B/s rd, 0 op/s 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:44.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:44.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:44.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:43 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 496 B/s rd, 0 op/s 2024-03-26T02:37:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:37:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:43 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:45.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:44 smithi164 ceph-mon[24851]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 438 B/s rd, 0 op/s 2024-03-26T02:37:45.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:44 smithi164 ceph-mon[24851]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-03-26T02:37:45.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:44 smithi164 ceph-mon[24851]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:37:45.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:44 smithi164 ceph-mon[24851]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:37:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:44 smithi164 ceph-mon[29512]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 438 B/s rd, 0 op/s 2024-03-26T02:37:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:44 smithi164 ceph-mon[29512]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-03-26T02:37:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:44 smithi164 ceph-mon[29512]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:37:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:44 smithi164 ceph-mon[29512]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:37:45.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:44 smithi172 ceph-mon[25688]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 438 B/s rd, 0 op/s 2024-03-26T02:37:45.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:44 smithi172 ceph-mon[25688]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 529 B/s rd, 0 op/s 2024-03-26T02:37:45.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:44 smithi172 ceph-mon[25688]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:37:45.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:44 smithi172 ceph-mon[25688]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:37:46.905 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:46 smithi164 ceph-mon[24851]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:46.905 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:46 smithi164 ceph-mon[29512]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:46 smithi172 ceph-mon[25688]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[29512]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:37:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:37:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:49.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[24851]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:49.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:49.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:49.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:37:49.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:37:49.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:48 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:48 smithi172 ceph-mon[25688]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:48 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:48 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:49.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:48 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:37:49.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:48 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:37:49.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:48 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:49 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:37:50.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:49 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:37:50.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:49 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/959043185' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:37:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:49 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:37:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:49 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:37:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:49 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/959043185' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:37:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:49 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:37:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:49 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:37:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:49 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/959043185' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:37:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:50 smithi164 ceph-mon[24851]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:50 smithi164 ceph-mon[29512]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:50 smithi172 ceph-mon[25688]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:37:52.256 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:37:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:37:52] "GET /metrics HTTP/1.1" 200 37992 "" "Prometheus/2.33.4" 2024-03-26T02:37:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:52 smithi164 ceph-mon[24851]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-03-26T02:37:53.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:52 smithi164 ceph-mon[29512]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-03-26T02:37:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:52 smithi172 ceph-mon[25688]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 105 B/s rd, 0 op/s 2024-03-26T02:37:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[24851]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 106 B/s rd, 0 op/s 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[24851]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 106 B/s rd, 0 op/s 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[29512]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 106 B/s rd, 0 op/s 2024-03-26T02:37:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[29512]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 106 B/s rd, 0 op/s 2024-03-26T02:37:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:54 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:54.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:54 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:54 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:54 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:54 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:54 smithi172 ceph-mon[25688]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 106 B/s rd, 0 op/s 2024-03-26T02:37:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:54 smithi172 ceph-mon[25688]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 106 B/s rd, 0 op/s 2024-03-26T02:37:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:54 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:37:54.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:54 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:56.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:56 smithi164 ceph-mon[24851]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:37:56.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:56 smithi164 ceph-mon[29512]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:37:56.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:56 smithi172 ceph-mon[25688]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:37:58.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:58 smithi164 ceph-mon[29512]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:37:58.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:58 smithi164 ceph-mon[24851]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:37:58.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:58 smithi172 ceph-mon[25688]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:37:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:37:59 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:59.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:37:59 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:37:59 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:00.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:00 smithi164 ceph-mon[29512]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:38:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:00 smithi164 ceph-mon[24851]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:38:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:00 smithi172 ceph-mon[25688]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 637 B/s rd, 0 op/s 2024-03-26T02:38:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:38:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:38:02] "GET /metrics HTTP/1.1" 200 37988 "" "Prometheus/2.33.4" 2024-03-26T02:38:02.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:02 smithi164 ceph-mon[24851]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:38:02.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:02 smithi164 ceph-mon[29512]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:38:02.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:02 smithi172 ceph-mon[25688]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:38:04.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:04 smithi164 ceph-mon[24851]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:38:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:04 smithi164 ceph-mon[29512]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:38:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:04 smithi172 ceph-mon[25688]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:38:06.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:06 smithi164 ceph-mon[24851]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:06.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:06 smithi164 ceph-mon[29512]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:06.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:06 smithi172 ceph-mon[25688]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:08.727 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:08 smithi164 ceph-mon[24851]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:08.727 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:08 smithi164 ceph-mon[29512]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:08.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:08 smithi172 ceph-mon[25688]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:09.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:09 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:09.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:09 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:09 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:09 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:09 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:09 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:10 smithi164 ceph-mon[24851]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:10.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:10 smithi164 ceph-mon[29512]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:10.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:10 smithi172 ceph-mon[25688]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:38:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:38:11] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-03-26T02:38:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:12 smithi164 ceph-mon[24851]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:12.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:12 smithi164 ceph-mon[29512]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:12.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:12 smithi172 ceph-mon[25688]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:14.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:14 smithi164 ceph-mon[24851]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:14.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:14 smithi164 ceph-mon[29512]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:14.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:14 smithi172 ceph-mon[25688]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:16.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:16 smithi164 ceph-mon[24851]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:16 smithi164 ceph-mon[29512]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:16 smithi172 ceph-mon[25688]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:18.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:18 smithi164 ceph-mon[24851]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:18.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:18 smithi164 ceph-mon[29512]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:18 smithi172 ceph-mon[25688]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:19 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:19.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:19 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:19 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:20.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:20 smithi164 ceph-mon[24851]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:20.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:20 smithi164 ceph-mon[29512]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:20 smithi172 ceph-mon[25688]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:38:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:38:21] "GET /metrics HTTP/1.1" 200 37993 "" "Prometheus/2.33.4" 2024-03-26T02:38:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:22 smithi164 ceph-mon[24851]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:22.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:22 smithi164 ceph-mon[29512]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:22.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:22 smithi172 ceph-mon[25688]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:24 smithi164 ceph-mon[24851]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:24 smithi164 ceph-mon[29512]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:24.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:24.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:24 smithi172 ceph-mon[25688]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:26 smithi164 ceph-mon[24851]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:26 smithi164 ceph-mon[29512]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:26.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:26 smithi172 ceph-mon[25688]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:28 smithi164 ceph-mon[24851]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:28.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:28 smithi164 ceph-mon[29512]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:28.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:28 smithi172 ceph-mon[25688]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:29 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:29 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:29 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:30 smithi164 ceph-mon[24851]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:30.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:30 smithi164 ceph-mon[29512]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:30 smithi172 ceph-mon[25688]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:38:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:38:31] "GET /metrics HTTP/1.1" 200 37994 "" "Prometheus/2.33.4" 2024-03-26T02:38:32.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:32 smithi164 ceph-mon[24851]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:32.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:32 smithi164 ceph-mon[29512]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:32 smithi172 ceph-mon[25688]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:34 smithi164 ceph-mon[24851]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:34.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:34 smithi164 ceph-mon[29512]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:34 smithi172 ceph-mon[25688]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:36 smithi164 ceph-mon[24851]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:36 smithi164 ceph-mon[29512]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:36 smithi172 ceph-mon[25688]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:38 smithi164 ceph-mon[24851]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:39.028 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:38 smithi164 ceph-mon[29512]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:39.029 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:38 smithi172 ceph-mon[25688]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:39.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:39 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:39.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:39 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:39.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:39 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:39.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:39 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:39.940 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:39 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:39.940 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:39 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:40 smithi172 ceph-mon[25688]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:40.849 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:40 smithi164 ceph-mon[24851]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:40.849 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:40 smithi164 ceph-mon[29512]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:38:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:38:41] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-26T02:38:42.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:42 smithi172 ceph-mon[25688]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:43.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:42 smithi164 ceph-mon[24851]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:43.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:42 smithi164 ceph-mon[29512]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:45.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:44 smithi164 ceph-mon[24851]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:45.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:44 smithi164 ceph-mon[29512]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:45.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:44 smithi172 ceph-mon[25688]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:47.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:46 smithi164 ceph-mon[24851]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:47.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:46 smithi164 ceph-mon[29512]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:46 smithi172 ceph-mon[25688]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:48 smithi164 ceph-mon[24851]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:48 smithi164 ceph-mon[29512]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:48 smithi172 ceph-mon[25688]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:49.958 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:49 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:49 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:49 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:50 smithi164 ceph-mon[24851]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:50.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:50 smithi164 ceph-mon[29512]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:51.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:50 smithi172 ceph-mon[25688]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:38:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:38:51] "GET /metrics HTTP/1.1" 200 37978 "" "Prometheus/2.33.4" 2024-03-26T02:38:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:52 smithi164 ceph-mon[24851]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:53.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:52 smithi164 ceph-mon[29512]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:52 smithi172 ceph-mon[25688]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:54.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:53 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:38:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:53 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:38:54.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:53 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:38:55.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:55.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:55.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:38:55.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:38:55.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:54 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:55.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:38:55.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:38:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:38:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:54 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:38:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:55 smithi164 ceph-mon[24851]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-26T02:38:56.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:55 smithi164 ceph-mon[24851]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 553 B/s rd, 0 op/s 2024-03-26T02:38:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:55 smithi164 ceph-mon[29512]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-26T02:38:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:55 smithi164 ceph-mon[29512]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 553 B/s rd, 0 op/s 2024-03-26T02:38:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:55 smithi172 ceph-mon[25688]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 909 B/s rd, 0 op/s 2024-03-26T02:38:56.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:55 smithi172 ceph-mon[25688]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 553 B/s rd, 0 op/s 2024-03-26T02:38:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:57 smithi164 ceph-mon[24851]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:38:58.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:57 smithi164 ceph-mon[29512]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:38:58.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:57 smithi172 ceph-mon[25688]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:38:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:59 smithi164 ceph-mon[24851]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:38:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:38:59 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:59 smithi164 ceph-mon[29512]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:38:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:38:59 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:59 smithi172 ceph-mon[25688]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:38:59.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:38:59 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:01.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:01 smithi164 ceph-mon[24851]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:39:01.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:01 smithi164 ceph-mon[29512]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:39:01.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:01 smithi172 ceph-mon[25688]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:39:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:39:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:39:01] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.33.4" 2024-03-26T02:39:03.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:03 smithi164 ceph-mon[24851]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:39:03.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:03 smithi164 ceph-mon[29512]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:39:03.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:03 smithi172 ceph-mon[25688]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:39:05.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:05 smithi164 ceph-mon[24851]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:39:05.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:05 smithi164 ceph-mon[29512]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:39:05.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:05 smithi172 ceph-mon[25688]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:39:07.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:07 smithi164 ceph-mon[24851]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:07.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:07 smithi164 ceph-mon[29512]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:07.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:07 smithi172 ceph-mon[25688]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:09 smithi164 ceph-mon[29512]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:09 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:09 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:09.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:09 smithi164 ceph-mon[24851]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:09.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:09 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:09.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:09 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:09 smithi172 ceph-mon[25688]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:09 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:09.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:09 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:11.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:11 smithi164 ceph-mon[24851]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:11.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:11 smithi164 ceph-mon[29512]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:11.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:11 smithi172 ceph-mon[25688]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:39:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:39:11] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-26T02:39:13.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:13 smithi164 ceph-mon[24851]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:13.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:13 smithi164 ceph-mon[29512]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:13.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:13 smithi172 ceph-mon[25688]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:15.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:15 smithi164 ceph-mon[24851]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:15.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:15 smithi164 ceph-mon[29512]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:15.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:15 smithi172 ceph-mon[25688]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:17.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:17 smithi164 ceph-mon[24851]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:17.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:17 smithi164 ceph-mon[29512]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:17 smithi172 ceph-mon[25688]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:19.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:19 smithi164 ceph-mon[29512]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:19.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:19 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:19 smithi164 ceph-mon[24851]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:19.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:19 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:19 smithi172 ceph-mon[25688]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:19 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:21 smithi164 ceph-mon[24851]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:21.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:21 smithi164 ceph-mon[29512]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:21 smithi172 ceph-mon[25688]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:39:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:39:21] "GET /metrics HTTP/1.1" 200 37986 "" "Prometheus/2.33.4" 2024-03-26T02:39:23.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:23 smithi164 ceph-mon[24851]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:23.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:23 smithi164 ceph-mon[29512]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:23.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:23 smithi172 ceph-mon[25688]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:24.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:24.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:24.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:25.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:25 smithi164 ceph-mon[24851]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:25.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:25 smithi164 ceph-mon[29512]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:25 smithi172 ceph-mon[25688]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:27.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:27 smithi164 ceph-mon[24851]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:27.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:27 smithi164 ceph-mon[29512]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:27.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:27 smithi172 ceph-mon[25688]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:29 smithi164 ceph-mon[29512]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:29 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:29.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:29 smithi164 ceph-mon[24851]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:29.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:29 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:29 smithi172 ceph-mon[25688]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:29 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:31.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:31 smithi164 ceph-mon[29512]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:31.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:31 smithi164 ceph-mon[24851]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:31.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:31 smithi172 ceph-mon[25688]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:39:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:39:31] "GET /metrics HTTP/1.1" 200 37989 "" "Prometheus/2.33.4" 2024-03-26T02:39:33.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:33 smithi164 ceph-mon[24851]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:33.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:33 smithi164 ceph-mon[29512]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:33.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:33 smithi172 ceph-mon[25688]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:35.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:35 smithi164 ceph-mon[29512]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:35.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:35 smithi164 ceph-mon[24851]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:35.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:35 smithi172 ceph-mon[25688]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:37.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:37 smithi164 ceph-mon[29512]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:37.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:37 smithi164 ceph-mon[24851]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:37.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:37 smithi172 ceph-mon[25688]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:39 smithi164 ceph-mon[24851]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:39 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:39.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:39 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:39 smithi164 ceph-mon[29512]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:39 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:39.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:39 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:39.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:39 smithi172 ceph-mon[25688]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:39.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:39 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:39.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:39 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:41.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:41 smithi172 ceph-mon[25688]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:41.999 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:41 smithi164 ceph-mon[24851]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:42.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:41 smithi164 ceph-mon[29512]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:39:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:39:41] "GET /metrics HTTP/1.1" 200 38003 "" "Prometheus/2.33.4" 2024-03-26T02:39:43.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:43 smithi172 ceph-mon[25688]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:43 smithi164 ceph-mon[24851]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:44.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:43 smithi164 ceph-mon[29512]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:45.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:45 smithi172 ceph-mon[25688]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:45 smithi164 ceph-mon[24851]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:45 smithi164 ceph-mon[29512]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:47.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:47 smithi172 ceph-mon[25688]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:48.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:47 smithi164 ceph-mon[24851]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:48.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:47 smithi164 ceph-mon[29512]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:50.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:49 smithi164 ceph-mon[29512]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:50.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:49 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:49 smithi164 ceph-mon[24851]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:50.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:49 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:49 smithi172 ceph-mon[25688]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:49 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:39:51.999 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:51 smithi164 ceph-mon[24851]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:52.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:51 smithi164 ceph-mon[29512]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:52.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:51 smithi172 ceph-mon[25688]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:39:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:39:51] "GET /metrics HTTP/1.1" 200 38003 "" "Prometheus/2.33.4" 2024-03-26T02:39:54.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:53 smithi164 ceph-mon[24851]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:53 smithi164 ceph-mon[29512]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:54.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:53 smithi172 ceph-mon[25688]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:54 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:54 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:39:55.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:54 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:55.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:54 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:39:55.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:54 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:39:55.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:54 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:39:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:55 smithi164 ceph-mon[29512]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:55 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:39:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:55 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:39:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:55 smithi164 ceph-mon[29512]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:39:56.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:55 smithi164 ceph-mon[24851]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:56.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:55 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:39:56.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:55 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:39:56.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:55 smithi164 ceph-mon[24851]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:39:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:55 smithi172 ceph-mon[25688]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:39:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:55 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:39:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:55 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:39:56.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:55 smithi172 ceph-mon[25688]: from='mgr.15150 ' entity='mgr.y' 2024-03-26T02:39:57.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:56 smithi164 ceph-mon[29512]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:39:57.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:56 smithi164 ceph-mon[29512]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:57.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:56 smithi164 ceph-mon[24851]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:39:57.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:56 smithi164 ceph-mon[24851]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:57.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:56 smithi172 ceph-mon[25688]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:39:57.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:56 smithi172 ceph-mon[25688]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:58 smithi164 ceph-mon[24851]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:59.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:58 smithi164 ceph-mon[29512]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:39:59.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:58 smithi172 ceph-mon[25688]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:00.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:39:59 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:00.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:39:59 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:00.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:39:59 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:00 smithi164 ceph-mon[24851]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:40:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:00 smithi164 ceph-mon[24851]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T02:40:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:00 smithi164 ceph-mon[24851]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T02:40:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:00 smithi164 ceph-mon[24851]: daemon grafana.a on smithi172 is in error state 2024-03-26T02:40:01.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:00 smithi164 ceph-mon[29512]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:40:01.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:00 smithi164 ceph-mon[29512]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T02:40:01.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:00 smithi164 ceph-mon[29512]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T02:40:01.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:00 smithi164 ceph-mon[29512]: daemon grafana.a on smithi172 is in error state 2024-03-26T02:40:01.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:00 smithi172 ceph-mon[25688]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:40:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:00 smithi172 ceph-mon[25688]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T02:40:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:00 smithi172 ceph-mon[25688]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T02:40:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:00 smithi172 ceph-mon[25688]: daemon grafana.a on smithi172 is in error state 2024-03-26T02:40:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:01 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:40:01] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.33.4" 2024-03-26T02:40:03.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:02 smithi164 ceph-mon[24851]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:03.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:02 smithi164 ceph-mon[29512]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:03.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:02 smithi172 ceph-mon[25688]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:05.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:04 smithi164 ceph-mon[24851]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:05.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:04 smithi164 ceph-mon[29512]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:05.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:04 smithi172 ceph-mon[25688]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:06 smithi164 ceph-mon[29512]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:40:07.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:06 smithi164 ceph-mon[24851]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:40:07.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:06 smithi172 ceph-mon[25688]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:40:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:08 smithi164 ceph-mon[24851]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:09.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:08 smithi164 ceph-mon[29512]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:08 smithi172 ceph-mon[25688]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:09.946 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:09 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:09.946 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:09 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:09.946 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:09 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:09.946 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:09 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:09 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:09 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:10 smithi164 ceph-mon[24851]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:10 smithi164 ceph-mon[29512]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:10 smithi172 ceph-mon[25688]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:40:11] "GET /metrics HTTP/1.1" 200 37994 "" "Prometheus/2.33.4" 2024-03-26T02:40:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:12 smithi164 ceph-mon[24851]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:13.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:12 smithi164 ceph-mon[29512]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:12 smithi172 ceph-mon[25688]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:14 smithi164 ceph-mon[24851]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:14 smithi164 ceph-mon[29512]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:14 smithi172 ceph-mon[25688]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:17.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:16 smithi164 ceph-mon[24851]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:17.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:16 smithi164 ceph-mon[29512]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:17.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:16 smithi172 ceph-mon[25688]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:18 smithi164 ceph-mon[24851]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:18 smithi164 ceph-mon[29512]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:18 smithi172 ceph-mon[25688]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:19.958 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:19 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:19 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:20.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:19 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:20 smithi164 ceph-mon[24851]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:20 smithi164 ceph-mon[29512]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:20 smithi172 ceph-mon[25688]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:40:21] "GET /metrics HTTP/1.1" 200 37994 "" "Prometheus/2.33.4" 2024-03-26T02:40:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:22 smithi164 ceph-mon[24851]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:22 smithi164 ceph-mon[29512]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:23.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:22 smithi172 ceph-mon[25688]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:24 smithi164 ceph-mon[24851]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:24 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:25.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:24 smithi164 ceph-mon[29512]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:25.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:24 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:25.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:24 smithi172 ceph-mon[25688]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:25.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:24 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:26.955 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:26 smithi164 ceph-mon[24851]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:26.955 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:26 smithi164 ceph-mon[29512]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:27.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:26 smithi172 ceph-mon[25688]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:27.325 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:40:28.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:28 smithi164 ceph-mon[24851]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:28.866 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:28 smithi164 ceph-mon[29512]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:29.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:28 smithi172 ceph-mon[25688]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (9m) 2m ago 17m 24.4M - 0.23.0 ba2b418f427c 98dbe984702c 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 error 2m ago 16m - - 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (2m) 2m ago 16m 50.0M - 3.5 e1d6a67b021e 1dfdc7cb0f53 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283 running (11m) 2m ago 21m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 d86222d71593 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (6m) 2m ago 23m 498M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (23m) 2m ago 23m 65.3M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (21m) 2m ago 21m 46.3M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:40:29.292 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (21m) 2m ago 21m 46.2M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (9m) 2m ago 17m 19.0M - 1.3.1 1dbe0e931976 547c321e8f73 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (8m) 2m ago 17m 17.1M - 1.3.1 1dbe0e931976 114ffb73b278 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (21m) 2m ago 21m 53.4M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (20m) 2m ago 20m 54.5M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (20m) 2m ago 20m 52.3M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (20m) 2m ago 20m 55.5M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (19m) 2m ago 19m 53.0M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (19m) 2m ago 19m 51.0M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (18m) 2m ago 18m 52.3M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (18m) 2m ago 18m 53.4M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (2m) 2m ago 17m 43.3M - 2.33.4 514e6a882f6e cc7530d80e7a 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (16m) 2m ago 16m 74.6M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:40:29.293 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (16m) 2m ago 16m 77.4M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:40:29.960 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions' 2024-03-26T02:40:30.986 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:30 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:30.986 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:30 smithi164 ceph-mon[24851]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:30.986 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:30 smithi164 ceph-mon[24851]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:40:30.986 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:30 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:30.986 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:30 smithi164 ceph-mon[29512]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:30.986 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:30 smithi164 ceph-mon[29512]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:40:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:30 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:30 smithi172 ceph-mon[25688]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:30 smithi172 ceph-mon[25688]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:40:32.061 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:40:32.062 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:40:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:40:31] "GET /metrics HTTP/1.1" 200 37997 "" "Prometheus/2.33.4" 2024-03-26T02:40:32.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:32 smithi164 ceph-mon[24851]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:32.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:32 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3538839112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:40:32.694 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:32 smithi164 ceph-mon[29512]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:32.694 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:32 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3538839112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:40:32.718 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph -s' 2024-03-26T02:40:33.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:32 smithi172 ceph-mon[25688]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:33.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:32 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3538839112' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:40:34.853 INFO:teuthology.orchestra.run.smithi164.stdout: cluster: 2024-03-26T02:40:34.853 INFO:teuthology.orchestra.run.smithi164.stdout: id: e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:40:34.853 INFO:teuthology.orchestra.run.smithi164.stdout: health: HEALTH_WARN 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: 1 failed cephadm daemon(s) 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: services: 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: mon: 3 daemons, quorum a,c,b (age 21m) 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: mgr: y(active, since 3m), standbys: x 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: osd: 8 osds: 8 up (since 18m), 8 in (since 18m) 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: rgw: 2 daemons active (2 hosts, 1 zones) 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: data: 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: pools: 6 pools, 161 pgs 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: objects: 241 objects, 585 KiB 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: usage: 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:34.854 INFO:teuthology.orchestra.run.smithi164.stdout: pgs: 161 active+clean 2024-03-26T02:40:34.855 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:40:34.855 INFO:teuthology.orchestra.run.smithi164.stdout: io: 2024-03-26T02:40:34.855 INFO:teuthology.orchestra.run.smithi164.stdout: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-26T02:40:34.855 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-03-26T02:40:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:34 smithi164 ceph-mon[24851]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:34 smithi164 ceph-mon[29512]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:34 smithi172 ceph-mon[25688]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:35.395 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:40:35.934 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:35 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2545904893' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:40:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:35 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2545904893' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:40:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:35 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2545904893' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T02:40:36.981 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:36 smithi164 ceph-mon[24851]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:36.982 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:36 smithi164 ceph-mon[29512]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:37.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:36 smithi172 ceph-mon[25688]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:37.496 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-03-26T02:40:37.496 INFO:teuthology.orchestra.run.smithi164.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T02:40:37.497 INFO:teuthology.orchestra.run.smithi164.stdout: daemon grafana.a on smithi172 is in error state 2024-03-26T02:40:37.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:37 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2770464321' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:40:37.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:37 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2770464321' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:40:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:37 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2770464321' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:40:38.143 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-26T02:40:38.790 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:38 smithi164 ceph-mon[24851]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:38.790 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:38 smithi164 ceph-mon[24851]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:38.791 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:38 smithi164 ceph-mon[29512]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:38.791 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:38 smithi164 ceph-mon[29512]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:38 smithi172 ceph-mon[25688]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:38 smithi172 ceph-mon[25688]: from='mgr.15150 172.21.15.164:0/3172726634' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:40.261 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:40:40.895 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:40 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:40.895 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:40 smithi164 ceph-mon[24851]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:40.895 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:40 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3210671335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:40:40.895 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:40 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:40.895 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:40 smithi164 ceph-mon[29512]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:40.895 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:40 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3210671335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:40:40.918 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph mgr fail' 2024-03-26T02:40:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:40 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:40 smithi172 ceph-mon[25688]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:40:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:40 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3210671335' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:40:42.290 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:40:41] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.33.4" 2024-03-26T02:40:42.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:42 smithi164 ceph-mon[29512]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:42 smithi164 ceph-mon[24851]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:42 smithi172 ceph-mon[25688]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:40:43.929 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3391056316' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:40:43.929 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-mon[25688]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T02:40:43.929 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-mon[25688]: Standby manager daemon y started 2024-03-26T02:40:43.930 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:40:43] ENGINE Bus STOPPING 2024-03-26T02:40:43.930 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:40:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:40:43.930 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:40:43] ENGINE Bus STOPPED 2024-03-26T02:40:43.930 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:40:43] ENGINE Bus STARTING 2024-03-26T02:40:43.930 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:40:43] ENGINE Serving on http://:::9283 2024-03-26T02:40:43.930 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:40:43] ENGINE Bus STARTED 2024-03-26T02:40:43.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3391056316' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:40:43.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-mon[24851]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T02:40:43.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-mon[24851]: Standby manager daemon y started 2024-03-26T02:40:43.933 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:43.693+0000 7ff719608700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-26T02:40:43.933 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ignoring --setuser ceph since I am not root 2024-03-26T02:40:43.933 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ignoring --setgroup ceph since I am not root 2024-03-26T02:40:43.933 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-mgr[62330]: -- 172.21.15.164:0/3316032636 <== mon.1 v2:172.21.15.164:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ed641185a0 con 0x55ed640ec800 2024-03-26T02:40:43.933 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3391056316' entity='client.admin' cmd=[{"prefix": "mgr fail"}]: dispatch 2024-03-26T02:40:43.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-mon[29512]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T02:40:43.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:43 smithi164 ceph-mon[29512]: Standby manager daemon y started 2024-03-26T02:40:44.387 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'sleep 180' 2024-03-26T02:40:44.687 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:44.295+0000 7f6a8cbfd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:40:44.688 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:44.395+0000 7f6a8cbfd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:40:44.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3391056316' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:40:44.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: mgrmap e31: x(active, starting, since 0.687317s), standbys: y 2024-03-26T02:40:44.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:40:44.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:40:44.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:40:44.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:40:44.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: Manager daemon x is now available 2024-03-26T02:40:44.939 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: [26/Mar/2024:02:40:44] ENGINE Bus STARTING 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: [26/Mar/2024:02:40:44] ENGINE Serving on https://172.21.15.172:7150 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: [26/Mar/2024:02:40:44] ENGINE Serving on http://172.21.15.172:8765 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[29512]: [26/Mar/2024:02:40:44] ENGINE Bus STARTED 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3391056316' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: mgrmap e31: x(active, starting, since 0.687317s), standbys: y 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:40:44.940 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:40:44.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: Manager daemon x is now available 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: [26/Mar/2024:02:40:44] ENGINE Bus STARTING 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: [26/Mar/2024:02:40:44] ENGINE Serving on https://172.21.15.172:7150 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: [26/Mar/2024:02:40:44] ENGINE Serving on http://172.21.15.172:8765 2024-03-26T02:40:44.942 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:44 smithi164 ceph-mon[24851]: [26/Mar/2024:02:40:44] ENGINE Bus STARTED 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3391056316' entity='client.admin' cmd='[{"prefix": "mgr fail"}]': finished 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: mgrmap e31: x(active, starting, since 0.687317s), standbys: y 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:40:44.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: Manager daemon x is now available 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-26T02:40:44.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: [26/Mar/2024:02:40:44] ENGINE Bus STARTING 2024-03-26T02:40:44.963 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: [26/Mar/2024:02:40:44] ENGINE Serving on https://172.21.15.172:7150 2024-03-26T02:40:44.963 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: [26/Mar/2024:02:40:44] ENGINE Serving on http://172.21.15.172:8765 2024-03-26T02:40:44.963 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:44 smithi172 ceph-mon[25688]: [26/Mar/2024:02:40:44] ENGINE Bus STARTED 2024-03-26T02:40:45.201 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:45.107+0000 7f6a8cbfd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:40:45.536 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:45.200+0000 7f6a8cbfd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:40:45.536 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:45.351+0000 7f6a8cbfd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:40:45.536 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:45.450+0000 7f6a8cbfd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:40:45.536 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:45.535+0000 7f6a8cbfd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:40:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-mon[24851]: mgrmap e32: x(active, since 1.69599s), standbys: y 2024-03-26T02:40:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-mon[24851]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:45.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-mon[29512]: mgrmap e32: x(active, since 1.69599s), standbys: y 2024-03-26T02:40:45.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-mon[29512]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:45 smithi172 ceph-mon[25688]: mgrmap e32: x(active, since 1.69599s), standbys: y 2024-03-26T02:40:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:45 smithi172 ceph-mon[25688]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:46.118 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:45.813+0000 7f6a8cbfd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:40:46.118 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:45.929+0000 7f6a8cbfd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:40:46.119 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:46.018+0000 7f6a8cbfd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:40:46.511 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:46.118+0000 7f6a8cbfd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:40:46.512 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:46.299+0000 7f6a8cbfd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:40:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-mon[24851]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:40:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-mon[24851]: Cluster is now healthy 2024-03-26T02:40:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-mon[24851]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:46.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-mon[29512]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:40:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-mon[29512]: Cluster is now healthy 2024-03-26T02:40:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:46 smithi164 ceph-mon[29512]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:46 smithi172 ceph-mon[25688]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:40:47.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:46 smithi172 ceph-mon[25688]: Cluster is now healthy 2024-03-26T02:40:47.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:46 smithi172 ceph-mon[25688]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:47.443 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:47 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:47.309+0000 7f6a8cbfd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:40:48.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:47 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:47.442+0000 7f6a8cbfd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:40:48.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:47 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:47.831+0000 7f6a8cbfd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:40:48.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:47 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:47.925+0000 7f6a8cbfd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:40:48.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:47 smithi164 ceph-mon[24851]: mgrmap e33: x(active, since 3s), standbys: y 2024-03-26T02:40:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:47 smithi164 ceph-mon[29512]: mgrmap e33: x(active, since 3s), standbys: y 2024-03-26T02:40:48.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:47 smithi172 ceph-mon[25688]: mgrmap e33: x(active, since 3s), standbys: y 2024-03-26T02:40:48.307 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:48.133+0000 7f6a8cbfd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:40:48.644 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:48.330+0000 7f6a8cbfd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:40:48.919 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-mon[24851]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:48.919 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:48.919 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:48.920 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:48.643+0000 7f6a8cbfd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:40:48.920 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-mon[29512]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:48.920 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:48.920 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:49.170 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:48 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:48.919+0000 7f6a8cbfd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:40:49.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:48 smithi172 ceph-mon[25688]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:49.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:48 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:49.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:48 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:49.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:49.170+0000 7f6a8cbfd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:40:50.208 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:49 smithi172 ceph-mon[25688]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:50.208 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:49 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:50.208 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:49 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:50.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:49 smithi164 ceph-mon[24851]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:50.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:49 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:50.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:49 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:50.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:50 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:50.037+0000 7f6a8cbfd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:40:50.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:49 smithi164 ceph-mon[29512]: from='client.25135 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:40:50.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:49 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:50.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:49 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:50.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:50 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:50.656+0000 7f6a8cbfd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:40:50.797 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:50 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:50.764+0000 7f6a8cbfd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:40:51.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:51.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:51.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: Standby manager daemon y restarted 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: Standby manager daemon y started 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:51.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: Standby manager daemon y restarted 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: Standby manager daemon y started 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:51.001+0000 7f6a8cbfd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:40:51] ENGINE Bus STARTING 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: CherryPy Checker: 2024-03-26T02:40:51.296 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: The Application mounted at '' has an empty config. 2024-03-26T02:40:51.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:40:51.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:40:51] ENGINE Serving on http://:::9283 2024-03-26T02:40:51.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:40:51] ENGINE Bus STARTED 2024-03-26T02:40:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:40:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: Standby manager daemon y restarted 2024-03-26T02:40:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: Standby manager daemon y started 2024-03-26T02:40:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-26T02:40:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:40:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-26T02:40:51.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:51 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:40:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:40:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:40:52.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: mgrmap e34: x(active, since 8s), standbys: y 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[24851]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: mgrmap e34: x(active, since 8s), standbys: y 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:52.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:40:52.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:40:52.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:40:52.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:40:52.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:40:52.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:52 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: mgrmap e34: x(active, since 8s), standbys: y 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:40:52.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:52 smithi172 ceph-mon[25688]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:40:53.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:40:53.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[24851]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:40:53.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:53 smithi164 ceph-mon[29512]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:40:53.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:40:53.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:40:53.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:40:53.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:40:53.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:40:53.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 28 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:40:53.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:40:53.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:40:53.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:40:53.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:40:53.559 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:40:53.560 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:40:53.560 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:53 smithi172 ceph-mon[25688]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:40:55.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:55 smithi164 ceph-mon[29512]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:55.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:55 smithi164 ceph-mon[24851]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:55.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-mon[25688]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:40:55.325+0000 7f960d783700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:40:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Creating ceph-iscsi config... 2024-03-26T02:40:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:40:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:40:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:40:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Traceback (most recent call last): 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: return _run_code(code, main_globals, None, 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: exec(code, run_globals) 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:40:55.808 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Traceback (most recent call last): 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2425, in _daemon_action 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: return self.event_loop.get_result(coro, timeout) 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-26T02:40:55.809 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: return future.result(timeout) 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: return self.__get_result() 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: raise self._exception 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: config_blobs=daemon_spec.final_config, 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Creating ceph-iscsi config... 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:40:55.810 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: Traceback (most recent call last): 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: return _run_code(code, main_globals, None, 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: exec(code, run_globals) 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:55.811 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:40:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:56 smithi164 ceph-mon[24851]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:40:56.794 INFO:journalctl@ceph.mon.a.smithi164.stdout: Creating ceph-iscsi config... 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:40:56.795 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2425, in _daemon_action 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: return future.result(timeout) 2024-03-26T02:40:56.796 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: return self.__get_result() 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: raise self._exception 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: Creating ceph-iscsi config... 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.797 INFO:journalctl@ceph.mon.a.smithi164.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:40:56.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:56 smithi164 ceph-mon[29512]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: Creating ceph-iscsi config... 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:40:56.799 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2425, in _daemon_action 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-26T02:40:56.800 INFO:journalctl@ceph.mon.c.smithi164.stdout: return future.result(timeout) 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: return self.__get_result() 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: raise self._exception 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: Creating ceph-iscsi config... 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:40:56.801 INFO:journalctl@ceph.mon.c.smithi164.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:40:56.802 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.a.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.a.smithi164.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.a.smithi164.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:56 smithi164 ceph-mon[24851]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:56 smithi164 ceph-mon[24851]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.803 INFO:journalctl@ceph.mon.c.smithi164.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:56 smithi164 ceph-mon[29512]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:40:56.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:56 smithi164 ceph-mon[29512]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:56 smithi172 ceph-mon[25688]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Creating ceph-iscsi config... 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:40:56.807 INFO:journalctl@ceph.mon.b.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: exec(code, run_globals) 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-03-26T02:40:56.808 INFO:journalctl@ceph.mon.b.smithi172.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2425, in _daemon_action 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: return self.event_loop.get_result(coro, timeout) 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: return future.result(timeout) 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: return self.__get_result() 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: raise self._exception 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1374, in _create_daemon 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: config_blobs=daemon_spec.final_config, 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1685, in _run_cephadm 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:40:56.809 INFO:journalctl@ceph.mon.b.smithi172.stdout: Creating ceph-iscsi config... 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Non-zero exit code 1 from systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: Traceback (most recent call last): 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: exec(code, run_globals) 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:40:56.810 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:40:56.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:40:56.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:40:56.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1123, in deploy_daemon 2024-03-26T02:40:56.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:40:56.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: RuntimeError: Failed command: systemctl restart ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:40:56.811 INFO:journalctl@ceph.mon.b.smithi172.stdout: See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:40:56.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:56 smithi172 ceph-mon[25688]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:40:56.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:56 smithi172 ceph-mon[25688]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:40:57.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:57 smithi172 ceph-mon[25688]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:57.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:57 smithi164 ceph-mon[24851]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:57.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:57 smithi164 ceph-mon[29512]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:58.883 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 systemd[1]: Stopping Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:40:58.883 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 bash[57858]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.882Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:40:59.142 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.883Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:40:59.143 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.883Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T02:40:59.143 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[55004]: ts=2024-03-26T02:40:58.883Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T02:40:59.143 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:58 smithi172 podman[57871]: 2024-03-26 02:40:58.892521265 +0000 UTC m=+0.129502729 container died cc7530d80e7a2fa5f52519c80711e1ba884f4cdfb5df9a6a3440dca8958130c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:40:59.424 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 podman[57871]: 2024-03-26 02:40:59.231422972 +0000 UTC m=+0.468404425 container cleanup cc7530d80e7a2fa5f52519c80711e1ba884f4cdfb5df9a6a3440dca8958130c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:40:59.424 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 bash[57871]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:40:59.677 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-mon[25688]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:59.677 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:59.678 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 podman[57885]: 2024-03-26 02:40:59.4240773 +0000 UTC m=+0.533685984 container remove cc7530d80e7a2fa5f52519c80711e1ba884f4cdfb5df9a6a3440dca8958130c8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:40:59.678 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 bash[57898]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:40:59.678 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T02:40:59.678 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 systemd[1]: Stopped Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:40:59.678 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.398s CPU time. 2024-03-26T02:40:59.678 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 systemd[1]: Starting Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:40:59.678 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 podman[57970]: 2024-03-26 02:40:59.658269902 +0000 UTC m=+0.096546993 container create d51f5e12df0adae074e1ed709842567dac0e29943fe8cf3ee421cc50aec42e2f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:40:59.678 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 podman[57970]: 2024-03-26 02:40:59.58065817 +0000 UTC m=+0.018935218 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T02:40:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:59 smithi164 ceph-mon[24851]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:40:59 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:59 smithi164 ceph-mon[29512]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:40:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:40:59 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:40:59.969 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 podman[57970]: 2024-03-26 02:40:59.885896385 +0000 UTC m=+0.324173433 container init d51f5e12df0adae074e1ed709842567dac0e29943fe8cf3ee421cc50aec42e2f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:40:59.969 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 podman[57970]: 2024-03-26 02:40:59.888413154 +0000 UTC m=+0.326690203 container start d51f5e12df0adae074e1ed709842567dac0e29943fe8cf3ee421cc50aec42e2f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:40:59.969 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.912Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T02:40:59.969 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.912Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.912Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.912Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi172 (none))" 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.912Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.912Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.913Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.914Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.914Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.917Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.917Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.1µs 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:40:59.917Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:40:59.970 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 bash[57970]: d51f5e12df0adae074e1ed709842567dac0e29943fe8cf3ee421cc50aec42e2f 2024-03-26T02:41:00.306 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:40:59 smithi172 systemd[1]: Started Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:41:00.307 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.118Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=4 2024-03-26T02:41:00.589 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.337Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=4 2024-03-26T02:41:00.589 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.538Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=4 2024-03-26T02:41:00.589 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.589Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=4 2024-03-26T02:41:00.589 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.589Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=4 2024-03-26T02:41:00.589 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.589Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=53.529µs wal_replay_duration=671.842631ms total_replay_duration=671.909387ms 2024-03-26T02:41:00.849 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.593Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:41:00.849 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.594Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T02:41:00.849 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.594Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:41:00.849 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.616Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.523516ms db_storage=837ns remote_storage=1.19µs web_handler=324ns query_engine=669ns scrape=373.171µs scrape_sd=98.982µs notify=11.06µs notify_sd=9.292µs rules=21.622372ms 2024-03-26T02:41:00.849 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:41:00 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:41:00.616Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T02:41:01.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:01.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:01.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[24851]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:41:01.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:01 smithi164 ceph-mon[29512]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:41:01.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:01 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:01.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:01 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:01.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:01 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:41:01.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:01 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:41:01.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:01 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:41:01.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:01 smithi172 ceph-mon[25688]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:41:03.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:03 smithi164 ceph-mon[24851]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-26T02:41:03.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:03 smithi164 ceph-mon[29512]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-26T02:41:03.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:03 smithi172 ceph-mon[25688]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 584 B/s rd, 0 op/s 2024-03-26T02:41:05.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:41:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:41:05.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:04 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 538 B/s rd, 0 op/s 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:41:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:04 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:05.805 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:05 smithi164 ceph-mon[24851]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:41:05.805 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:05 smithi164 ceph-mon[24851]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:41:05.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:05 smithi164 ceph-mon[29512]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:41:05.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:05 smithi164 ceph-mon[29512]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:41:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:05 smithi172 ceph-mon[25688]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:41:06.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:05 smithi172 ceph-mon[25688]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:41:06.661 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:06 smithi164 ceph-mon[24851]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:06.661 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:06 smithi164 ceph-mon[29512]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:06 smithi172 ceph-mon[25688]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:07.977 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:07 smithi164 ceph-mon[24851]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:07.977 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:07 smithi164 ceph-mon[29512]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:07 smithi172 ceph-mon[25688]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:09.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/804047188' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2711962715' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]: dispatch 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]: dispatch 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:09.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:41:09.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:41:09.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:41:09.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:41:09.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:41:09.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/804047188' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:41:09.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2711962715' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]: dispatch 2024-03-26T02:41:09.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:09 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]: dispatch 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/804047188' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2711962715' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]: dispatch 2024-03-26T02:41:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:09 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]: dispatch 2024-03-26T02:41:11.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]': finished 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[29512]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[29512]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2692762674' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]: dispatch 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]: dispatch 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]': finished 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[24851]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[24851]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2692762674' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]: dispatch 2024-03-26T02:41:11.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:10 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]: dispatch 2024-03-26T02:41:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:10 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1295084522"}]': finished 2024-03-26T02:41:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:10 smithi172 ceph-mon[25688]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T02:41:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:10 smithi172 ceph-mon[25688]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:41:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:10 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2692762674' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]: dispatch 2024-03-26T02:41:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:10 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]: dispatch 2024-03-26T02:41:11.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:11 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]': finished 2024-03-26T02:41:11.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:11 smithi164 ceph-mon[29512]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T02:41:11.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:11 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2214605462' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3722022543"}]: dispatch 2024-03-26T02:41:11.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:11 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]': finished 2024-03-26T02:41:11.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:11 smithi164 ceph-mon[24851]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T02:41:11.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:11 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2214605462' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3722022543"}]: dispatch 2024-03-26T02:41:12.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:11 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/4166607833"}]': finished 2024-03-26T02:41:12.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:11 smithi172 ceph-mon[25688]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T02:41:12.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:11 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2214605462' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3722022543"}]: dispatch 2024-03-26T02:41:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:12 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2214605462' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3722022543"}]': finished 2024-03-26T02:41:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:12 smithi164 ceph-mon[24851]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T02:41:13.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:12 smithi164 ceph-mon[24851]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T02:41:13.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:12 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2408792219' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/4077812041"}]: dispatch 2024-03-26T02:41:13.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:12 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2214605462' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3722022543"}]': finished 2024-03-26T02:41:13.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:12 smithi164 ceph-mon[29512]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T02:41:13.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:12 smithi164 ceph-mon[29512]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T02:41:13.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:12 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2408792219' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/4077812041"}]: dispatch 2024-03-26T02:41:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:12 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2214605462' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3722022543"}]': finished 2024-03-26T02:41:13.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:12 smithi172 ceph-mon[25688]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T02:41:13.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:12 smithi172 ceph-mon[25688]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T02:41:13.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:12 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2408792219' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/4077812041"}]: dispatch 2024-03-26T02:41:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2408792219' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/4077812041"}]': finished 2024-03-26T02:41:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T02:41:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1680609037' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]: dispatch 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]: dispatch 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2408792219' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/4077812041"}]': finished 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1680609037' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]: dispatch 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]: dispatch 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:14.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:41:14.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:13 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2408792219' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/4077812041"}]': finished 2024-03-26T02:41:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T02:41:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1680609037' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]: dispatch 2024-03-26T02:41:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]: dispatch 2024-03-26T02:41:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:41:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:41:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:13 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:41:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[24851]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:41:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[24851]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-03-26T02:41:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]': finished 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[24851]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/121495922' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]: dispatch 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]: dispatch 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[29512]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[29512]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]': finished 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[29512]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/121495922' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]: dispatch 2024-03-26T02:41:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:14 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]: dispatch 2024-03-26T02:41:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:14 smithi172 ceph-mon[25688]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:41:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:14 smithi172 ceph-mon[25688]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 595 B/s rd, 0 op/s 2024-03-26T02:41:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:14 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/4077812041"}]': finished 2024-03-26T02:41:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:14 smithi172 ceph-mon[25688]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T02:41:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:14 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:14 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/121495922' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]: dispatch 2024-03-26T02:41:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:14 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]: dispatch 2024-03-26T02:41:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:15 smithi172 ceph-mon[25688]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]': finished 2024-03-26T02:41:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:15 smithi172 ceph-mon[25688]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T02:41:15.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:41:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:41:15] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.33.4" 2024-03-26T02:41:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:15 smithi164 ceph-mon[24851]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]': finished 2024-03-26T02:41:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:15 smithi164 ceph-mon[24851]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T02:41:16.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:15 smithi164 ceph-mon[29512]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/3121102719"}]': finished 2024-03-26T02:41:16.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:15 smithi164 ceph-mon[29512]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T02:41:17.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:16 smithi164 ceph-mon[24851]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:17.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:16 smithi164 ceph-mon[29512]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:17.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:16 smithi172 ceph-mon[25688]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:18 smithi164 ceph-mon[24851]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-26T02:41:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:18 smithi164 ceph-mon[29512]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-26T02:41:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:18 smithi172 ceph-mon[25688]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 918 B/s rd, 0 op/s 2024-03-26T02:41:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:20 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:20 smithi164 ceph-mon[24851]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:41:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:20 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:20 smithi164 ceph-mon[29512]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:41:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:20 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:21.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:20 smithi172 ceph-mon[25688]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:41:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:22 smithi164 ceph-mon[24851]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:41:23.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:22 smithi164 ceph-mon[29512]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:41:23.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:22 smithi172 ceph-mon[25688]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:41:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:24 smithi164 ceph-mon[24851]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-26T02:41:25.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:24 smithi164 ceph-mon[29512]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-26T02:41:25.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:24 smithi172 ceph-mon[25688]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 0 op/s 2024-03-26T02:41:25.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:41:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:41:25] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.33.4" 2024-03-26T02:41:27.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:26 smithi164 ceph-mon[24851]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:41:27.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:26 smithi164 ceph-mon[29512]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:41:27.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:26 smithi172 ceph-mon[25688]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:41:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:27 smithi164 ceph-mon[24851]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:27 smithi164 ceph-mon[29512]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:28.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:27 smithi172 ceph-mon[25688]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:29 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:29 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:29 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:30 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:30 smithi164 ceph-mon[24851]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:30 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:30.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:30 smithi164 ceph-mon[29512]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:30 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:30.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:30 smithi172 ceph-mon[25688]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:32.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:32 smithi164 ceph-mon[24851]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:32.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:32 smithi164 ceph-mon[29512]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:32 smithi172 ceph-mon[25688]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:34.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:34 smithi164 ceph-mon[24851]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:34.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:34 smithi164 ceph-mon[29512]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:34.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:34 smithi172 ceph-mon[25688]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:35.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:41:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:41:35] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.33.4" 2024-03-26T02:41:36.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:36 smithi164 ceph-mon[24851]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:36.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:36 smithi164 ceph-mon[29512]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:36.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:36 smithi172 ceph-mon[25688]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:38.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:38 smithi164 ceph-mon[24851]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:38.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:38 smithi164 ceph-mon[29512]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:38.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:38 smithi172 ceph-mon[25688]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:40 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:40.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:40 smithi164 ceph-mon[24851]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:40 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:40 smithi164 ceph-mon[29512]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:40 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:40.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:40 smithi172 ceph-mon[25688]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:42 smithi164 ceph-mon[24851]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:42.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:42 smithi164 ceph-mon[29512]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:42.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:42 smithi172 ceph-mon[25688]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:44 smithi164 ceph-mon[24851]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:44 smithi164 ceph-mon[29512]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:44.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:44 smithi172 ceph-mon[25688]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:41:45.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:41:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:41:45] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.33.4" 2024-03-26T02:41:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:46 smithi164 ceph-mon[24851]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:46.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:46 smithi164 ceph-mon[29512]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:46 smithi172 ceph-mon[25688]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:48.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:48 smithi172 ceph-mon[25688]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:48 smithi164 ceph-mon[24851]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:48 smithi164 ceph-mon[29512]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:50 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:50 smithi172 ceph-mon[25688]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:51.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:50 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:51.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:50 smithi164 ceph-mon[24851]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:50 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:41:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:50 smithi164 ceph-mon[29512]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:52.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:52 smithi172 ceph-mon[25688]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:52.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:52 smithi164 ceph-mon[24851]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:52.807 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:52 smithi164 ceph-mon[29512]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:54.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:54 smithi172 ceph-mon[25688]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:54 smithi164 ceph-mon[24851]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:55.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:54 smithi164 ceph-mon[29512]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:41:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:41:55] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.33.4" 2024-03-26T02:41:57.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:56 smithi164 ceph-mon[24851]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:57.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:56 smithi164 ceph-mon[29512]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:57.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:56 smithi172 ceph-mon[25688]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:41:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:58 smithi164 ceph-mon[24851]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:59.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:58 smithi164 ceph-mon[29512]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:41:59.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:58 smithi172 ceph-mon[25688]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:00.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:41:59 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:00.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:41:59 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:00.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:41:59 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:00 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:00 smithi164 ceph-mon[24851]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:01.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:00 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:01.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:00 smithi164 ceph-mon[29512]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:01.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:00 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:01.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:00 smithi172 ceph-mon[25688]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:03.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:02 smithi164 ceph-mon[24851]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:03.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:02 smithi164 ceph-mon[29512]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:03.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:02 smithi172 ceph-mon[25688]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:05.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:04 smithi164 ceph-mon[24851]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:05.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:04 smithi164 ceph-mon[29512]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:05.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:04 smithi172 ceph-mon[25688]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:05.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:42:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:42:05] "GET /metrics HTTP/1.1" 200 37996 "" "Prometheus/2.33.4" 2024-03-26T02:42:07.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:06 smithi164 ceph-mon[24851]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:06 smithi164 ceph-mon[29512]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:07.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:06 smithi172 ceph-mon[25688]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:08 smithi164 ceph-mon[24851]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:09.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:08 smithi164 ceph-mon[29512]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:08 smithi172 ceph-mon[25688]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:10.846 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:10 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:10.847 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:10 smithi164 ceph-mon[29512]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:10.847 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:10 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:10.847 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:10 smithi164 ceph-mon[24851]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:10 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:10 smithi172 ceph-mon[25688]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:12 smithi164 ceph-mon[24851]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:13.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:12 smithi164 ceph-mon[29512]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:12 smithi172 ceph-mon[25688]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:13 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:42:14.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:13 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:42:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:13 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:42:15.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[29512]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:15.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:15.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:42:15.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:42:15.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:42:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[24851]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:42:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:42:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:14 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:42:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:14 smithi172 ceph-mon[25688]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:14 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:14 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:42:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:14 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:42:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:14 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:42:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:15 smithi172 ceph-mon[25688]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-03-26T02:42:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:15 smithi172 ceph-mon[25688]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-26T02:42:15.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:42:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:42:15] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.33.4" 2024-03-26T02:42:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:15 smithi164 ceph-mon[24851]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-03-26T02:42:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:15 smithi164 ceph-mon[24851]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-26T02:42:16.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:15 smithi164 ceph-mon[29512]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 947 B/s rd, 0 op/s 2024-03-26T02:42:16.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:15 smithi164 ceph-mon[29512]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 581 B/s rd, 0 op/s 2024-03-26T02:42:18.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:17 smithi164 ceph-mon[24851]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:17 smithi164 ceph-mon[29512]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:17 smithi172 ceph-mon[25688]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:19 smithi164 ceph-mon[24851]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:20.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:19 smithi164 ceph-mon[29512]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:19 smithi172 ceph-mon[25688]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:20.958 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:20 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:20 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:20 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:22.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:21 smithi164 ceph-mon[24851]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:22.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:21 smithi164 ceph-mon[29512]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:22.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:21 smithi172 ceph-mon[25688]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:23 smithi164 ceph-mon[24851]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:24.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:23 smithi164 ceph-mon[29512]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:24.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:23 smithi172 ceph-mon[25688]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:42:25.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:25 smithi172 ceph-mon[25688]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:42:25.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:42:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:42:25] "GET /metrics HTTP/1.1" 200 37990 "" "Prometheus/2.33.4" 2024-03-26T02:42:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:25 smithi164 ceph-mon[24851]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:42:26.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:25 smithi164 ceph-mon[29512]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:42:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:27 smithi164 ceph-mon[24851]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:27 smithi164 ceph-mon[29512]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:28.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:27 smithi172 ceph-mon[25688]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:29 smithi164 ceph-mon[24851]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:29 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:30.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:29 smithi164 ceph-mon[29512]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:30.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:29 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:30.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:29 smithi172 ceph-mon[25688]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:30.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:29 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:30 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:30 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:30 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:32.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:31 smithi164 ceph-mon[24851]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:32.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:31 smithi164 ceph-mon[29512]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:32.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:31 smithi172 ceph-mon[25688]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:33 smithi164 ceph-mon[24851]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:34.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:33 smithi164 ceph-mon[29512]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:34.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:33 smithi172 ceph-mon[25688]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:35.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:35 smithi172 ceph-mon[25688]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:35.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:42:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:42:35] "GET /metrics HTTP/1.1" 200 37992 "" "Prometheus/2.33.4" 2024-03-26T02:42:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:35 smithi164 ceph-mon[24851]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:36.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:35 smithi164 ceph-mon[29512]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:37 smithi164 ceph-mon[24851]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:38.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:37 smithi164 ceph-mon[29512]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:37 smithi172 ceph-mon[25688]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:39 smithi164 ceph-mon[24851]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:39 smithi164 ceph-mon[29512]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:39 smithi172 ceph-mon[25688]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:40.946 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:40 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:40.946 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:40 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:40 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:42.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:41 smithi164 ceph-mon[24851]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:42.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:41 smithi164 ceph-mon[29512]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:42.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:41 smithi172 ceph-mon[25688]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:42 smithi164 ceph-mon[24851]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:42 smithi164 ceph-mon[29512]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:42 smithi172 ceph-mon[25688]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:44.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:44.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:44.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:42:45.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:45 smithi164 ceph-mon[24851]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:45.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:45 smithi164 ceph-mon[29512]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:45.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:45 smithi172 ceph-mon[25688]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:45.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:42:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:42:45] "GET /metrics HTTP/1.1" 200 37992 "" "Prometheus/2.33.4" 2024-03-26T02:42:47.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:47 smithi164 ceph-mon[24851]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:47.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:47 smithi164 ceph-mon[29512]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:47.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:47 smithi172 ceph-mon[25688]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:49 smithi164 ceph-mon[24851]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:49.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:49 smithi164 ceph-mon[29512]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:49 smithi172 ceph-mon[25688]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:50.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:50 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:50.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:50 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:50 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:42:51.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:51 smithi164 ceph-mon[24851]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:51 smithi164 ceph-mon[29512]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:51 smithi172 ceph-mon[25688]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:53.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:53 smithi164 ceph-mon[24851]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:53.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:53 smithi164 ceph-mon[29512]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:53.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:53 smithi172 ceph-mon[25688]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:55.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:55 smithi172 ceph-mon[25688]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:42:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:42:55] "GET /metrics HTTP/1.1" 200 37992 "" "Prometheus/2.33.4" 2024-03-26T02:42:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:55 smithi164 ceph-mon[24851]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:55 smithi164 ceph-mon[29512]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:42:57.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:57 smithi172 ceph-mon[25688]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:57 smithi164 ceph-mon[24851]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:42:58.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:57 smithi164 ceph-mon[29512]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:00.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:59 smithi164 ceph-mon[24851]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:00.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:42:59 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:00.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:59 smithi164 ceph-mon[29512]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:00.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:42:59 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:00.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:59 smithi172 ceph-mon[25688]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:00.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:42:59 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:00 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:01.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:00 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:01.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:00 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:02.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:01 smithi164 ceph-mon[29512]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:02.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:01 smithi164 ceph-mon[24851]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:02.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:01 smithi172 ceph-mon[25688]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:04.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:03 smithi164 ceph-mon[24851]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:04.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:03 smithi164 ceph-mon[29512]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:04.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:03 smithi172 ceph-mon[25688]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:05.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:05 smithi172 ceph-mon[25688]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:05.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:43:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:43:05] "GET /metrics HTTP/1.1" 200 38005 "" "Prometheus/2.33.4" 2024-03-26T02:43:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:05 smithi164 ceph-mon[24851]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:05 smithi164 ceph-mon[29512]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:08.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:07 smithi164 ceph-mon[24851]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:08.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:07 smithi164 ceph-mon[29512]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:07 smithi172 ceph-mon[25688]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:10.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:09 smithi164 ceph-mon[24851]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:10.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:09 smithi164 ceph-mon[29512]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:10.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:09 smithi172 ceph-mon[25688]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:10 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:10 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:10 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:12.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:11 smithi164 ceph-mon[24851]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:12.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:11 smithi164 ceph-mon[29512]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:12.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:11 smithi172 ceph-mon[25688]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:14.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:13 smithi164 ceph-mon[29512]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:13 smithi164 ceph-mon[24851]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:13 smithi172 ceph-mon[25688]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:43:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:43:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:43:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:43:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:43:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:43:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:43:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:14 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:43:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:14 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:14 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:43:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:14 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:43:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:14 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:43:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:14 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:43:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:15 smithi172 ceph-mon[25688]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:15 smithi172 ceph-mon[25688]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-26T02:43:15.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:15 smithi172 ceph-mon[25688]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:43:15.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:43:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:43:15] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.33.4" 2024-03-26T02:43:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:15 smithi164 ceph-mon[24851]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:15 smithi164 ceph-mon[24851]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-26T02:43:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:15 smithi164 ceph-mon[24851]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:43:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:15 smithi164 ceph-mon[29512]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:15 smithi164 ceph-mon[29512]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-03-26T02:43:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:15 smithi164 ceph-mon[29512]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:43:18.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:17 smithi164 ceph-mon[24851]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:17 smithi164 ceph-mon[29512]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:17 smithi172 ceph-mon[25688]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:19.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:19 smithi164 ceph-mon[24851]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:43:19.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:19 smithi164 ceph-mon[29512]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:43:19.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:19 smithi172 ceph-mon[25688]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 589 B/s rd, 0 op/s 2024-03-26T02:43:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:20 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:20.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:20 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:20.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:20 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:21 smithi164 ceph-mon[24851]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:21.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:21 smithi164 ceph-mon[29512]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:21 smithi172 ceph-mon[25688]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:23 smithi164 ceph-mon[24851]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:24.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:23 smithi164 ceph-mon[29512]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:24.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:23 smithi172 ceph-mon[25688]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:25.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:25 smithi172 ceph-mon[25688]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:43:25.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:43:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:43:25] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.33.4" 2024-03-26T02:43:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:25 smithi164 ceph-mon[24851]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:43:26.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:25 smithi164 ceph-mon[29512]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:43:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:27 smithi164 ceph-mon[24851]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:27 smithi164 ceph-mon[29512]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:28.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:27 smithi172 ceph-mon[25688]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:29 smithi164 ceph-mon[24851]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:29 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:29.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:29 smithi164 ceph-mon[29512]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:29.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:29 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:29.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:29 smithi172 ceph-mon[25688]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:29.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:29 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:30.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:30 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:30.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:30 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:30.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:30 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:31.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:31 smithi172 ceph-mon[25688]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:32.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:31 smithi164 ceph-mon[24851]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:32.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:31 smithi164 ceph-mon[29512]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:33 smithi164 ceph-mon[24851]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:33 smithi164 ceph-mon[29512]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:34.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:33 smithi172 ceph-mon[25688]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:35.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:35 smithi172 ceph-mon[25688]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:35.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:43:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:43:35] "GET /metrics HTTP/1.1" 200 38003 "" "Prometheus/2.33.4" 2024-03-26T02:43:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:35 smithi164 ceph-mon[24851]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:35 smithi164 ceph-mon[29512]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:37 smithi164 ceph-mon[24851]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:37 smithi164 ceph-mon[29512]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:37 smithi172 ceph-mon[25688]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:39.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:39 smithi172 ceph-mon[25688]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:39 smithi164 ceph-mon[24851]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:39 smithi164 ceph-mon[29512]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:40.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:40 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:40 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:41.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:40 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:41.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:41 smithi172 ceph-mon[25688]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:42.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:41 smithi164 ceph-mon[24851]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:42.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:41 smithi164 ceph-mon[29512]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:43 smithi164 ceph-mon[24851]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:43 smithi164 ceph-mon[29512]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:43 smithi172 ceph-mon[25688]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:45.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:44 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:45.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:44 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:45.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:44 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:43:45.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:45 smithi172 ceph-mon[25688]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:45.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:43:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:43:45] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.33.4" 2024-03-26T02:43:46.030 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:45 smithi164 ceph-mon[29512]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:46.030 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:45 smithi164 ceph-mon[24851]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:46.735 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:43:47.987 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:47 smithi164 ceph-mon[29512]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:47.988 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:47 smithi164 ceph-mon[24851]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:48.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:47 smithi172 ceph-mon[25688]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:48.715 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (12m) 2m ago 20m 24.2M - 0.23.0 ba2b418f427c 98dbe984702c 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 error 2m ago 20m - - 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (2m) 2m ago 19m 56.3M - 3.5 e1d6a67b021e fa60ba06a6dd 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283 running (15m) 2m ago 24m 497M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 d86222d71593 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (9m) 2m ago 26m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (26m) 2m ago 26m 70.9M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (25m) 2m ago 25m 53.3M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (25m) 2m ago 25m 51.0M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (12m) 2m ago 21m 18.7M - 1.3.1 1dbe0e931976 547c321e8f73 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (12m) 2m ago 21m 17.2M - 1.3.1 1dbe0e931976 114ffb73b278 2024-03-26T02:43:48.716 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (24m) 2m ago 24m 53.6M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (24m) 2m ago 24m 54.7M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (23m) 2m ago 23m 52.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (23m) 2m ago 23m 55.6M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (23m) 2m ago 23m 53.4M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (22m) 2m ago 22m 51.3M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (22m) 2m ago 22m 52.9M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (21m) 2m ago 21m 53.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (2m) 2m ago 21m 44.0M - 2.33.4 514e6a882f6e d51f5e12df0a 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (19m) 2m ago 19m 75.6M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:43:48.717 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (19m) 2m ago 19m 78.2M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:43:49.404 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ls' 2024-03-26T02:43:49.663 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:49 smithi164 ceph-mon[24851]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:49.663 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:49 smithi164 ceph-mon[29512]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:49 smithi172 ceph-mon[25688]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:50.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:50 smithi172 ceph-mon[25688]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:43:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:50 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:50.852 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:50 smithi164 ceph-mon[29512]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:43:50.852 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:50 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:50.853 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:50 smithi164 ceph-mon[24851]: from='client.15369 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:43:50.853 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:50 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:43:51.434 INFO:teuthology.orchestra.run.smithi164.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-26T02:43:51.435 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager ?:9093,9094 1/1 2m ago 21m smithi164=a;count:1 2024-03-26T02:43:51.435 INFO:teuthology.orchestra.run.smithi164.stdout:grafana ?:3000 0/1 2m ago 21m smithi172=a;count:1 2024-03-26T02:43:51.435 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo ?:5000 1/1 2m ago 19m count:1 2024-03-26T02:43:51.435 INFO:teuthology.orchestra.run.smithi164.stdout:mgr 2/2 2m ago 24m smithi164=y;smithi172=x;count:2 2024-03-26T02:43:51.435 INFO:teuthology.orchestra.run.smithi164.stdout:mon 3/3 2m ago 25m smithi164:172.21.15.164=a;smithi164:[v2:172.21.15.164:3301,v1:172.21.15.164:6790]=c;smithi172:172.21.15.172=b;count:3 2024-03-26T02:43:51.435 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter ?:9100 2/2 2m ago 21m smithi164=a;smithi172=b;count:2 2024-03-26T02:43:51.436 INFO:teuthology.orchestra.run.smithi164.stdout:osd 8 2m ago - 2024-03-26T02:43:51.436 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus ?:9095 1/1 2m ago 21m smithi172=a;count:1 2024-03-26T02:43:51.436 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo ?:8000 2/2 2m ago 19m count:2 2024-03-26T02:43:51.766 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:51 smithi164 ceph-mon[24851]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:51.766 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:51 smithi164 ceph-mon[29512]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:51 smithi172 ceph-mon[25688]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:52.049 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions' 2024-03-26T02:43:52.608 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:52 smithi164 ceph-mon[24851]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:43:52.608 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:52 smithi164 ceph-mon[29512]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:43:53.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:52 smithi172 ceph-mon[25688]: from='client.25282 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:43:54.038 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:53 smithi164 ceph-mon[24851]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:43:54.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:43:54.040 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:43:54.040 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:53 smithi164 ceph-mon[29512]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:54.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:53 smithi172 ceph-mon[25688]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:54.657 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:54 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1063510038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:43:54.657 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:54 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1063510038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:43:54.682 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-26T02:43:55.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:54 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1063510038' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:43:55.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:55 smithi164 ceph-mon[24851]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:55.690 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:55 smithi164 ceph-mon[29512]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:55.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:55 smithi172 ceph-mon[25688]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:43:55.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:43:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:43:55] "GET /metrics HTTP/1.1" 200 38002 "" "Prometheus/2.33.4" 2024-03-26T02:43:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:57 smithi164 ceph-mon[24851]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:58.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:57 smithi164 ceph-mon[29512]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:58.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:57 smithi172 ceph-mon[25688]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:43:59.022 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:58 smithi164 ceph-mon[24851]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:43:59.023 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:58 smithi164 ceph-mon[29512]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:43:59.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:58 smithi172 ceph-mon[25688]: from='client.15387 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:00.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:59 smithi164 ceph-mon[29512]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:00.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:43:59 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:00.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:59 smithi164 ceph-mon[24851]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:00.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:43:59 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:00.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:59 smithi172 ceph-mon[25688]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:00.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:43:59 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:00.189 INFO:teuthology.orchestra.run.smithi164.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:44:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:00 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:00.825 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-26T02:44:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:00 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:00 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:00 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:00 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:00 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:00 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:02.008 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:01 smithi164 ceph-mon[29512]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:44:02.008 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:01 smithi164 ceph-mon[29512]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:44:02.009 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:01 smithi164 ceph-mon[29512]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:44:02.009 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:01 smithi164 ceph-mon[29512]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:44:02.009 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:01 smithi164 ceph-mon[24851]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:44:02.009 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:01 smithi164 ceph-mon[24851]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:44:02.009 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:01 smithi164 ceph-mon[24851]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:44:02.009 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:01 smithi164 ceph-mon[24851]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:44:02.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:01 smithi172 ceph-mon[25688]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:44:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:01 smithi172 ceph-mon[25688]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:44:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:01 smithi172 ceph-mon[25688]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:44:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:01 smithi172 ceph-mon[25688]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:44:03.486 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:44:04.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:03 smithi164 ceph-mon[24851]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:44:04.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:03 smithi164 ceph-mon[29512]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:44:04.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:03 smithi172 ceph-mon[25688]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:44:04.146 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:44:04.146 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (12m) 2m ago 20m 24.2M - 0.23.0 ba2b418f427c 98dbe984702c 2024-03-26T02:44:04.146 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 error 3m ago 20m - - 2024-03-26T02:44:04.146 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (2m) 2m ago 19m 56.3M - 3.5 e1d6a67b021e fa60ba06a6dd 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283 running (15m) 3m ago 25m 497M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 d86222d71593 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (9m) 2m ago 26m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (26m) 2m ago 26m 70.9M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (25m) 3m ago 25m 53.3M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (25m) 2m ago 25m 51.0M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (12m) 2m ago 21m 18.7M - 1.3.1 1dbe0e931976 547c321e8f73 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (12m) 3m ago 21m 17.2M - 1.3.1 1dbe0e931976 114ffb73b278 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (24m) 2m ago 24m 53.6M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (24m) 2m ago 24m 54.7M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (24m) 2m ago 24m 52.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (23m) 2m ago 23m 55.6M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (23m) 3m ago 23m 53.4M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (22m) 3m ago 22m 51.3M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:44:04.147 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (22m) 3m ago 22m 52.9M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:44:04.148 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (22m) 3m ago 22m 53.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:44:04.148 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (3m) 3m ago 21m 44.0M - 2.33.4 514e6a882f6e d51f5e12df0a 2024-03-26T02:44:04.148 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (19m) 2m ago 19m 75.6M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:44:04.148 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (19m) 3m ago 19m 78.2M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:44:04.535 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:44:04.536 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:44:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[29512]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:04.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[29512]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[29512]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/596479151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[24851]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[24851]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[24851]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:44:04.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/596479151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:04.919 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:04.863+0000 7f967b114700 -1 mgr handle_mgr_map I was active but no longer am 2024-03-26T02:44:04.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-mon[25688]: from='client.15393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:04.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' 2024-03-26T02:44:04.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:04.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-mon[25688]: from='mgr.34922 172.21.15.172:0/964612198' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T02:44:04.919 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-03-26T02:44:04.920 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-mon[25688]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:44:04.920 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/596479151' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:05.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:44:04] ENGINE Bus STOPPING 2024-03-26T02:44:05.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:44:04] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:44:05.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:44:04] ENGINE Bus STOPPED 2024-03-26T02:44:05.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:04 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:44:04] ENGINE Bus STARTING 2024-03-26T02:44:05.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:44:05] ENGINE Serving on http://:::9283 2024-03-26T02:44:05.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:44:05] ENGINE Bus STARTED 2024-03-26T02:44:05.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ignoring --setuser ceph since I am not root 2024-03-26T02:44:05.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ignoring --setgroup ceph since I am not root 2024-03-26T02:44:05.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:04 smithi172 ceph-mgr[45916]: -- 172.21.15.172:0/3628395730 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a7700fc5a0 con 0x55a7700d1000 2024-03-26T02:44:05.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:05.450+0000 7f5562890200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:44:05.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:05.547+0000 7f5562890200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:44:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:44:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:44:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T02:44:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T02:44:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: Failing over to other MGR 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.34922 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: mgrmap e35: y(active, starting, since 0.995493s) 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:44:05.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: Manager daemon y is now available 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:44:05.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:44:05.886 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:44:05.886 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:44:05.886 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:44:05.886 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-26T02:44:05.886 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [ 2024-03-26T02:44:05.886 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr" 2024-03-26T02:44:05.887 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:44:05.887 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "2/2 daemons upgraded", 2024-03-26T02:44:05.887 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:44:05.887 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:44:05.887 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:44:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: Failing over to other MGR 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.34922 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: mgrmap e35: y(active, starting, since 0.995493s) 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:44:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: Manager daemon y is now available 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:06.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='client.25306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: Upgrade: Need to upgrade myself (mgr.x) 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: Failing over to other MGR 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 669 B/s rd, 0 op/s 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.34922 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-03-26T02:44:06.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: mgrmap e35: y(active, starting, since 0.995493s) 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:44:06.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:44:06.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: Manager daemon y is now available 2024-03-26T02:44:06.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:06.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:06.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:44:06.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:44:06.327 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:06.237+0000 7f5562890200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:44:06.587 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:06.332+0000 7f5562890200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:44:06.587 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:06.476+0000 7f5562890200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:44:06.587 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:06.586+0000 7f5562890200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:44:06.899 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-mon[25688]: [26/Mar/2024:02:44:05] ENGINE Bus STARTING 2024-03-26T02:44:06.899 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-mon[25688]: [26/Mar/2024:02:44:05] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:44:06.900 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-mon[25688]: [26/Mar/2024:02:44:05] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:44:06.900 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-mon[25688]: [26/Mar/2024:02:44:05] ENGINE Bus STARTED 2024-03-26T02:44:06.900 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-mon[25688]: mgrmap e36: y(active, since 2s) 2024-03-26T02:44:06.900 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:06.675+0000 7f5562890200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[24851]: [26/Mar/2024:02:44:05] ENGINE Bus STARTING 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[24851]: [26/Mar/2024:02:44:05] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[24851]: [26/Mar/2024:02:44:05] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[24851]: [26/Mar/2024:02:44:05] ENGINE Bus STARTED 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[24851]: mgrmap e36: y(active, since 2s) 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[29512]: [26/Mar/2024:02:44:05] ENGINE Bus STARTING 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[29512]: [26/Mar/2024:02:44:05] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[29512]: [26/Mar/2024:02:44:05] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[29512]: [26/Mar/2024:02:44:05] ENGINE Bus STARTED 2024-03-26T02:44:06.928 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:06 smithi164 ceph-mon[29512]: mgrmap e36: y(active, since 2s) 2024-03-26T02:44:07.159 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:06 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:06.956+0000 7f5562890200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:44:07.159 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:07.070+0000 7f5562890200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:44:07.160 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:07.158+0000 7f5562890200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:44:07.464 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:07.266+0000 7f5562890200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:44:07.464 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:07.463+0000 7f5562890200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:44:07.774 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-mon[25688]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:07.774 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-mon[25688]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:07.774 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-mon[25688]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:44:07.775 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-mon[25688]: Cluster is now healthy 2024-03-26T02:44:07.775 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:07 smithi172 ceph-mon[25688]: mgrmap e37: y(active, since 3s) 2024-03-26T02:44:07.967 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[29512]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:07.967 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[29512]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:07.967 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[29512]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:44:07.967 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[29512]: Cluster is now healthy 2024-03-26T02:44:07.967 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[29512]: mgrmap e37: y(active, since 3s) 2024-03-26T02:44:07.967 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[24851]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:07.967 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[24851]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:07.968 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[24851]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:44:07.968 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[24851]: Cluster is now healthy 2024-03-26T02:44:07.968 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:07 smithi164 ceph-mon[24851]: mgrmap e37: y(active, since 3s) 2024-03-26T02:44:08.674 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:08 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:08.461+0000 7f5562890200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:44:08.674 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:08 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:08.595+0000 7f5562890200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:44:08.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:08 smithi164 ceph-mon[24851]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:08.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:08 smithi164 ceph-mon[29512]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:08.950 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:08 smithi172 ceph-mon[25688]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:09.284 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:08 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:08.950+0000 7f5562890200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:44:09.285 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:09 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:09.069+0000 7f5562890200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:44:09.285 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:09 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:09.283+0000 7f5562890200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:44:09.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:09 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:09.482+0000 7f5562890200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:44:10.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:09 smithi164 ceph-mon[29512]: mgrmap e38: y(active, since 4s) 2024-03-26T02:44:10.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:09 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:10.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:09 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:10.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:09 smithi164 ceph-mon[24851]: mgrmap e38: y(active, since 4s) 2024-03-26T02:44:10.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:09 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:10.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:09 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:10.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:09 smithi172 ceph-mon[25688]: mgrmap e38: y(active, since 4s) 2024-03-26T02:44:10.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:09 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:10.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:09 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:10.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:09 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:09.795+0000 7f5562890200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:44:10.329 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:10 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:10.067+0000 7f5562890200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:44:11.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[29512]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:11.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[29512]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:11.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[24851]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:11.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[24851]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:11.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:10 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:10 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:10.328+0000 7f5562890200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:44:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:10 smithi172 ceph-mon[25688]: from='client.35024 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:10 smithi172 ceph-mon[25688]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:10 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:10 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:10 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:10 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:11.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:11.208+0000 7f5562890200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:44:12.169 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:11.834+0000 7f5562890200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:44:12.169 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:11.941+0000 7f5562890200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:44:12.169 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:12.168+0000 7f5562890200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:44:12.170 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:11 smithi172 ceph-mon[25688]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:44:12.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:12.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:12.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[24851]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:44:12.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:44:12.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:11 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:44:12.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:44:12] ENGINE Bus STARTING 2024-03-26T02:44:12.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: CherryPy Checker: 2024-03-26T02:44:12.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: The Application mounted at '' has an empty config. 2024-03-26T02:44:12.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: 2024-03-26T02:44:12.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:44:12] ENGINE Serving on http://:::9283 2024-03-26T02:44:12.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: [26/Mar/2024:02:44:12] ENGINE Bus STARTED 2024-03-26T02:44:13.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 90 op/s 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: Standby manager daemon x started 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:44:13.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:44:13.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:44:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 90 op/s 2024-03-26T02:44:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:44:13.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:44:13.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:13.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:44:13.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: Standby manager daemon x started 2024-03-26T02:44:13.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:44:13.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:44:13.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:44:13.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:12 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 90 op/s 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: Reconfiguring iscsi.foo.smithi164.bueptz (dependencies changed)... 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: Reconfiguring daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:44:13.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: Standby manager daemon x started 2024-03-26T02:44:13.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:44:13.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:44:13.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:44:13.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:12 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/2004688013' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:44:14.183 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:13 smithi164 ceph-mon[24851]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:44:14.183 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:13 smithi164 ceph-mon[24851]: mgrmap e39: y(active, since 9s), standbys: x 2024-03-26T02:44:14.183 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:44:14.184 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:13 smithi164 ceph-mon[29512]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:44:14.184 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:13 smithi164 ceph-mon[29512]: mgrmap e39: y(active, since 9s), standbys: x 2024-03-26T02:44:14.184 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:44:14.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:13 smithi172 ceph-mon[25688]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T02:44:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:13 smithi172 ceph-mon[25688]: mgrmap e39: y(active, since 9s), standbys: x 2024-03-26T02:44:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:44:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:14 smithi164 ceph-mon[24851]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 68 op/s 2024-03-26T02:44:15.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:14 smithi164 ceph-mon[29512]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 68 op/s 2024-03-26T02:44:15.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:14 smithi172 ceph-mon[25688]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 0 B/s wr, 68 op/s 2024-03-26T02:44:15.806 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[45912]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:44:15] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-03-26T02:44:17.725 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:17 smithi172 ceph-mon[25688]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 154 op/s 2024-03-26T02:44:17.725 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:17 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:17.725 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:17 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:17.725 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:17 smithi172 ceph-mon[25688]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:44:17.725 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:17 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/355807015' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:44:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[29512]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 154 op/s 2024-03-26T02:44:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[29512]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:44:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/355807015' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:44:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[24851]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 106 KiB/s rd, 0 B/s wr, 154 op/s 2024-03-26T02:44:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:18.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:18.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[24851]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:44:18.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:17 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/355807015' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:44:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:18 smithi172 ceph-mon[25688]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:44:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:18 smithi164 ceph-mon[29512]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:44:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:18 smithi164 ceph-mon[24851]: Reconfiguring daemon prometheus.a on smithi172 2024-03-26T02:44:19.904 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:19 smithi172 ceph-mon[25688]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:19 smithi164 ceph-mon[24851]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:20.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:19 smithi164 ceph-mon[29512]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:20.306 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 systemd[1]: Stopping Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:44:20.307 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 bash[60725]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:44:20.594 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.325Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.326Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.326Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[57985]: ts=2024-03-26T02:44:20.326Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T02:44:20.595 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 podman[60739]: 2024-03-26 02:44:20.335837932 +0000 UTC m=+0.265653036 container died d51f5e12df0adae074e1ed709842567dac0e29943fe8cf3ee421cc50aec42e2f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:20.967 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:20 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:20.968 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 podman[60739]: 2024-03-26 02:44:20.691426198 +0000 UTC m=+0.621241300 container cleanup d51f5e12df0adae074e1ed709842567dac0e29943fe8cf3ee421cc50aec42e2f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:20.968 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 bash[60739]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:44:20.968 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 bash[60765]: Error: no container with name or ID "ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus.a" found: no such container 2024-03-26T02:44:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:20 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:20 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:21.227 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 podman[60753]: 2024-03-26 02:44:20.967598709 +0000 UTC m=+0.634204185 container remove d51f5e12df0adae074e1ed709842567dac0e29943fe8cf3ee421cc50aec42e2f (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:21.227 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T02:44:21.227 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 systemd[1]: Stopped Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:44:21.227 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.425s CPU time. 2024-03-26T02:44:21.227 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:20 smithi172 systemd[1]: Starting Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:44:21.227 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 podman[60837]: 2024-03-26 02:44:21.113401668 +0000 UTC m=+0.019485872 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-03-26T02:44:21.227 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 podman[60837]: 2024-03-26 02:44:21.226893543 +0000 UTC m=+0.132977736 container create 35f9aa6bb3b6c2304d218a0cca91cfc8bb82ac9d8389dd8fcfd879a3377089b9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 podman[60837]: 2024-03-26 02:44:21.437376222 +0000 UTC m=+0.343460416 container init 35f9aa6bb3b6c2304d218a0cca91cfc8bb82ac9d8389dd8fcfd879a3377089b9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 podman[60837]: 2024-03-26 02:44:21.440619583 +0000 UTC m=+0.346703785 container start 35f9aa6bb3b6c2304d218a0cca91cfc8bb82ac9d8389dd8fcfd879a3377089b9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.464Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.464Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.464Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.464Z caller=main.go:518 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi172 (none))" 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.464Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.464Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.465Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:44:21.521 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.466Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-03-26T02:44:21.522 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.467Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-03-26T02:44:21.522 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.470Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:44:21.522 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.470Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.878µs 2024-03-26T02:44:21.522 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.470Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:44:21.522 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 bash[60837]: 35f9aa6bb3b6c2304d218a0cca91cfc8bb82ac9d8389dd8fcfd879a3377089b9 2024-03-26T02:44:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:44:21.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:21.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 systemd[1]: Started Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:44:21.808 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.671Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=5 2024-03-26T02:44:22.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:22.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:44:22.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:21 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:22.233 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:21.941Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=5 2024-03-26T02:44:22.233 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.174Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=5 2024-03-26T02:44:22.233 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.233Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=5 2024-03-26T02:44:22.507 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.236Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=5 2024-03-26T02:44:22.508 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.236Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=5 2024-03-26T02:44:22.508 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.236Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=73.449µs wal_replay_duration=766.425081ms total_replay_duration=766.523852ms 2024-03-26T02:44:22.508 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.241Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:44:22.508 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.241Z caller=main.go:947 level=info msg="TSDB started" 2024-03-26T02:44:22.508 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.241Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:44:22.508 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.261Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.169299ms db_storage=712ns remote_storage=1.272µs web_handler=316ns query_engine=747ns scrape=420.559µs scrape_sd=105.109µs notify=20.438µs notify_sd=15.666µs rules=19.274396ms 2024-03-26T02:44:22.508 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:44:22.261Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-03-26T02:44:22.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:44:22.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-mon[25688]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:44:22.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:44:22.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:44:22.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:22 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:44:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:44:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[24851]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:44:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:44:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:44:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:44:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-03-26T02:44:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[29512]: Adding iSCSI gateway http://:@172.21.15.164:5000 to Dashboard 2024-03-26T02:44:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-03-26T02:44:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi164"}]: dispatch 2024-03-26T02:44:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:22 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:44:23.947 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:23 smithi172 ceph-mon[25688]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:23.947 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:23 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:44:23.947 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:23 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:44:23.947 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:23 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:23 smithi164 ceph-mon[24851]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:23 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:44:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:23 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:44:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:23 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:23 smithi164 ceph-mon[29512]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 97 KiB/s rd, 0 B/s wr, 140 op/s 2024-03-26T02:44:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:23 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:44:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:23 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:44:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:23 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:24.675 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:24 smithi172 ceph-mon[25688]: Upgrade: Updating mgr.x 2024-03-26T02:44:24.676 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:24 smithi172 ceph-mon[25688]: Deploying daemon mgr.x on smithi172 2024-03-26T02:44:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:24 smithi164 ceph-mon[24851]: Upgrade: Updating mgr.x 2024-03-26T02:44:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:24 smithi164 ceph-mon[24851]: Deploying daemon mgr.x on smithi172 2024-03-26T02:44:25.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:24 smithi164 ceph-mon[29512]: Upgrade: Updating mgr.x 2024-03-26T02:44:25.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:24 smithi164 ceph-mon[29512]: Deploying daemon mgr.x on smithi172 2024-03-26T02:44:25.261 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:24 smithi172 systemd[1]: Stopping Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:44:25.547 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:25 smithi172 podman[61186]: 2024-03-26 02:44:25.26107088 +0000 UTC m=+0.181973106 container died d86222d71593381cdcd0e0363f2dd66b5b4a5767cb06f3d3da98706a53622aba (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T02:44:25.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:25 smithi172 ceph-mon[25688]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-26T02:44:25.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:25 smithi172 podman[61186]: 2024-03-26 02:44:25.629799377 +0000 UTC m=+0.550701611 container cleanup d86222d71593381cdcd0e0363f2dd66b5b4a5767cb06f3d3da98706a53622aba (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:44:25.807 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:25 smithi172 bash[61186]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x 2024-03-26T02:44:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:25 smithi164 ceph-mon[24851]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-26T02:44:26.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:25 smithi164 ceph-mon[29512]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-26T02:44:26.232 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:25 smithi172 podman[61200]: 2024-03-26 02:44:25.906114646 +0000 UTC m=+0.642774103 container remove d86222d71593381cdcd0e0363f2dd66b5b4a5767cb06f3d3da98706a53622aba (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T02:44:26.232 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:25 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:44:26.232 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service: Failed with result 'exit-code'. 2024-03-26T02:44:26.232 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 systemd[1]: Stopped Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:44:26.232 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service: Consumed 57.326s CPU time. 2024-03-26T02:44:26.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 systemd[1]: Starting Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:44:26.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 podman[61333]: 2024-03-26 02:44:26.441033189 +0000 UTC m=+0.108890027 container create fe45bef004bdcc5389f2e5408df19ca8f6ecff4169f5144d9b407916dd4f9b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:44:26.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 podman[61333]: 2024-03-26 02:44:26.350960595 +0000 UTC m=+0.018817391 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:44:27.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 podman[61333]: 2024-03-26 02:44:26.668292625 +0000 UTC m=+0.336149408 container init fe45bef004bdcc5389f2e5408df19ca8f6ecff4169f5144d9b407916dd4f9b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T02:44:27.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 podman[61333]: 2024-03-26 02:44:26.670986829 +0000 UTC m=+0.338843612 container start fe45bef004bdcc5389f2e5408df19ca8f6ecff4169f5144d9b407916dd4f9b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:44:27.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 ceph-mgr[61353]: -- 172.21.15.172:0/3786933215 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5641085945a0 con 0x564108568800 2024-03-26T02:44:27.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 bash[61333]: fe45bef004bdcc5389f2e5408df19ca8f6ecff4169f5144d9b407916dd4f9b92 2024-03-26T02:44:27.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:26 smithi172 systemd[1]: Started Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:44:27.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:27.236+0000 7fad3b449200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:44:27.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:27.336+0000 7fad3b449200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:44:27.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:27 smithi172 ceph-mon[25688]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-26T02:44:27.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:27 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:27.906 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:27 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:27.907 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:27 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:27.907 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:27 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:27.923 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[24851]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-26T02:44:27.923 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:27.923 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:27.923 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:27.923 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:27.923 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[29512]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 0 B/s wr, 84 op/s 2024-03-26T02:44:27.923 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:27.923 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:27.924 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:27.924 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:27 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:28.272 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:28.024+0000 7fad3b449200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:44:28.272 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:28.117+0000 7fad3b449200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:44:28.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:28.271+0000 7fad3b449200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:44:28.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:28.367+0000 7fad3b449200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:44:28.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:28.453+0000 7fad3b449200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:44:28.842 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:28.730+0000 7fad3b449200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:44:28.842 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:28.841+0000 7fad3b449200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:44:29.262 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:28.947+0000 7fad3b449200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:44:29.262 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:29.066+0000 7fad3b449200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:44:29.262 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:29.261+0000 7fad3b449200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:44:29.975 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:29 smithi164 ceph-mon[24851]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:29.975 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:29 smithi164 ceph-mon[29512]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:30.049 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:29 smithi172 ceph-mon[25688]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:30.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:30.268+0000 7fad3b449200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:44:30.786 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:30.402+0000 7fad3b449200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:44:30.786 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:30.785+0000 7fad3b449200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:44:31.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:30.882+0000 7fad3b449200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:44:31.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:31.095+0000 7fad3b449200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:44:31.557 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:31.295+0000 7fad3b449200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:44:31.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-mon[25688]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:31.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:31.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:31.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:31.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:31.887 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:31.604+0000 7fad3b449200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:44:31.887 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:31.885+0000 7fad3b449200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:44:32.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[24851]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[29512]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:31 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:44:32] "GET /metrics HTTP/1.1" 200 38000 "" "Prometheus/2.33.4" 2024-03-26T02:44:32.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:32 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:32.147+0000 7fad3b449200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:44:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:32 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:32 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:32 smithi164 ceph-mon[24851]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:32 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:32 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:32 smithi164 ceph-mon[29512]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:32 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:33.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:32 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:33.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:32 smithi172 ceph-mon[25688]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:33.307 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:33.020+0000 7fad3b449200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:44:33.713 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:33.607+0000 7fad3b449200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:44:33.714 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:33.712+0000 7fad3b449200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:44:34.058 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:33.945+0000 7fad3b449200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:44:34.058 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: [26/Mar/2024:02:44:33] ENGINE Bus STARTING 2024-03-26T02:44:34.490 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: CherryPy Checker: 2024-03-26T02:44:34.490 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: The Application mounted at '' has an empty config. 2024-03-26T02:44:34.490 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:44:34.490 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: [26/Mar/2024:02:44:34] ENGINE Serving on http://:::9283 2024-03-26T02:44:34.490 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: [26/Mar/2024:02:44:34] ENGINE Bus STARTED 2024-03-26T02:44:34.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-mon[25688]: Standby manager daemon x restarted 2024-03-26T02:44:34.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-mon[25688]: Standby manager daemon x started 2024-03-26T02:44:34.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:44:34.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:44:34.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:44:34.792 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:34 smithi172 ceph-mon[25688]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:44:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[24851]: Standby manager daemon x restarted 2024-03-26T02:44:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[24851]: Standby manager daemon x started 2024-03-26T02:44:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:44:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:44:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:44:35.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[24851]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:44:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[29512]: Standby manager daemon x restarted 2024-03-26T02:44:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[29512]: Standby manager daemon x started 2024-03-26T02:44:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:44:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:44:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:44:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:34 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3200960969' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:44:35.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:35.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: mgrmap e40: y(active, since 30s), standbys: x 2024-03-26T02:44:35.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:35.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: mgrmap e40: y(active, since 30s), standbys: x 2024-03-26T02:44:35.933 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:35.935 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: mgrmap e40: y(active, since 30s), standbys: x 2024-03-26T02:44:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:36.254 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:44:36.905 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-26T02:44:36.905 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-26T02:44:36.905 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all mgr 2024-03-26T02:44:36.906 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all crash 2024-03-26T02:44:36.906 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all mds 2024-03-26T02:44:36.906 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:44:36.906 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:44:36.906 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:44:36.907 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all nfs 2024-03-26T02:44:36.907 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:44:36.907 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-26T02:44:36.907 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-26T02:44:36.907 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all mgr 2024-03-26T02:44:36.908 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all crash 2024-03-26T02:44:36.908 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all mds 2024-03-26T02:44:36.908 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:44:36.908 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:44:36.908 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:44:36.908 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all nfs 2024-03-26T02:44:36.908 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:36 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (13m) 5s ago 21m 24.0M - 0.23.0 ba2b418f427c 98dbe984702c 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 error 6s ago 20m - - 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (20s) 5s ago 20m 44.1M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (10s) 6s ago 25m 66.5M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (10m) 5s ago 27m 497M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (27m) 5s ago 27m 76.1M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (25m) 6s ago 25m 57.8M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (26m) 5s ago 26m 55.1M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (13m) 5s ago 21m 18.9M - 1.3.1 1dbe0e931976 547c321e8f73 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (12m) 6s ago 21m 19.9M - 1.3.1 1dbe0e931976 114ffb73b278 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (25m) 5s ago 25m 54.9M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:44:36.925 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (25m) 5s ago 24m 56.5M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (24m) 5s ago 24m 53.4M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (24m) 5s ago 24m 57.2M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (23m) 6s ago 23m 56.1M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (23m) 6s ago 23m 52.3M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (23m) 6s ago 23m 54.7M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (22m) 6s ago 22m 55.3M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (15s) 6s ago 21m 53.1M - 2.33.4 514e6a882f6e 35f9aa6bb3b6 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (20m) 5s ago 20m 76.7M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:44:36.926 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (20m) 6s ago 20m 79.2M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:44:37.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-26T02:44:37.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 558 B/s rd, 0 op/s 2024-03-26T02:44:37.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all mgr 2024-03-26T02:44:37.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all crash 2024-03-26T02:44:37.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all mds 2024-03-26T02:44:37.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:44:37.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:44:37.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:44:37.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all nfs 2024-03-26T02:44:37.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:36 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:44:37.339 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:44:37.339 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:44:37.339 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:44:37.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:44:37.339 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:44:37.339 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:44:37.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:44:37.339 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:44:37.340 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:44:37.716 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:37 smithi164 ceph-mon[24851]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:37.716 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:37 smithi164 ceph-mon[24851]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:37.716 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:37 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:37.716 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:37 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3707486869' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:37.716 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:37 smithi164 ceph-mon[29512]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:37.716 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:37 smithi164 ceph-mon[29512]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:37.717 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:37 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:37.717 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:37 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3707486869' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [ 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr" 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "2/2 daemons upgraded", 2024-03-26T02:44:37.717 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading node-exporter daemons", 2024-03-26T02:44:37.718 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:44:37.718 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:44:37.996 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:37 smithi164 systemd[1]: Stopping Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:44:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:37 smithi172 ceph-mon[25688]: from='client.15480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:37 smithi172 ceph-mon[25688]: from='client.25390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:37 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:37 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3707486869' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:38.294 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:37 smithi164 podman[76913]: 2024-03-26 02:44:37.994989622 +0000 UTC m=+0.128537309 container died 547c321e8f73a49e95f2c6fa9b7524060cf48a7f30151907dc25f2d280d809de (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:38.573 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:38 smithi164 podman[76913]: 2024-03-26 02:44:38.309691117 +0000 UTC m=+0.443238814 container cleanup 547c321e8f73a49e95f2c6fa9b7524060cf48a7f30151907dc25f2d280d809de (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:38.573 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:38 smithi164 bash[76913]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T02:44:38.573 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:38 smithi164 podman[76927]: 2024-03-26 02:44:38.552674321 +0000 UTC m=+0.558290353 container remove 547c321e8f73a49e95f2c6fa9b7524060cf48a7f30151907dc25f2d280d809de (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:38.574 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:38 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:44:38.988 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:38 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T02:44:38.989 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:38 smithi164 systemd[1]: Stopped Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:44:38.989 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:38 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 5.212s CPU time. 2024-03-26T02:44:38.989 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[24851]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:38.989 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[24851]: Upgrade: Updating node-exporter.a (1/2) 2024-03-26T02:44:38.989 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[24851]: Deploying daemon node-exporter.a on smithi164 2024-03-26T02:44:38.990 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[24851]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:38.990 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[24851]: from='client.25408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:38.990 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[29512]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:38.990 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[29512]: Upgrade: Updating node-exporter.a (1/2) 2024-03-26T02:44:38.990 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[29512]: Deploying daemon node-exporter.a on smithi164 2024-03-26T02:44:38.990 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[29512]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:38.991 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:38 smithi164 ceph-mon[29512]: from='client.25408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:38 smithi172 ceph-mon[25688]: from='client.25396 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:38 smithi172 ceph-mon[25688]: Upgrade: Updating node-exporter.a (1/2) 2024-03-26T02:44:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:38 smithi172 ceph-mon[25688]: Deploying daemon node-exporter.a on smithi164 2024-03-26T02:44:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:38 smithi172 ceph-mon[25688]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:38 smithi172 ceph-mon[25688]: from='client.25408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:44:39.294 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:38 smithi164 systemd[1]: Starting Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:44:39.294 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:39 smithi164 bash[77054]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T02:44:40.044 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:39 smithi164 bash[77054]: Getting image source signatures 2024-03-26T02:44:40.044 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:39 smithi164 bash[77054]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T02:44:40.044 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:39 smithi164 bash[77054]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T02:44:40.044 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:39 smithi164 bash[77054]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T02:44:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:40 smithi164 ceph-mon[24851]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:40.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:40 smithi164 ceph-mon[29512]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:40 smithi172 ceph-mon[25688]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:41.526 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:41 smithi164 bash[77054]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T02:44:41.794 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:41 smithi164 bash[77054]: Writing manifest to image destination 2024-03-26T02:44:42.099 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:44:42] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.33.4" 2024-03-26T02:44:42.099 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:41 smithi164 podman[77054]: 2024-03-26 02:44:41.773337375 +0000 UTC m=+2.688027146 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T02:44:42.099 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:41 smithi164 podman[77054]: 2024-03-26 02:44:41.880299095 +0000 UTC m=+2.794988866 container create 467a30dd8758ea35788b6db7fc2d5b4942ee0245ae33ededc5c9c0df065e1a0c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:42.100 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 podman[77054]: 2024-03-26 02:44:42.099335003 +0000 UTC m=+3.014024774 container init 467a30dd8758ea35788b6db7fc2d5b4942ee0245ae33ededc5c9c0df065e1a0c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 podman[77054]: 2024-03-26 02:44:42.102269949 +0000 UTC m=+3.016959723 container start 467a30dd8758ea35788b6db7fc2d5b4942ee0245ae33ededc5c9c0df065e1a0c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.105Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.105Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.106Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.106Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.106Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.106Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T02:44:42.545 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T02:44:42.546 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T02:44:42.546 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T02:44:42.546 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T02:44:42.546 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T02:44:42.546 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T02:44:42.546 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T02:44:42.546 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T02:44:42.546 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T02:44:42.547 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T02:44:42.548 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.107Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.108Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a[77117]: ts=2024-03-26T02:44:42.108Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 bash[77054]: 467a30dd8758ea35788b6db7fc2d5b4942ee0245ae33ededc5c9c0df065e1a0c 2024-03-26T02:44:42.549 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:44:42 smithi164 systemd[1]: Started Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:44:42.876 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:42 smithi172 ceph-mon[25688]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:42.876 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:42 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:42.876 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:42 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-mon[24851]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-mon[29512]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:42 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:44.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:44 smithi172 ceph-mon[25688]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:44.807 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:44 smithi172 systemd[1]: Stopping Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:44:45.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:44 smithi164 ceph-mon[24851]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:45.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:44 smithi164 ceph-mon[29512]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:44:45.134 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:44 smithi172 podman[62699]: 2024-03-26 02:44:44.812055424 +0000 UTC m=+0.106549228 container died 114ffb73b2787362f22fc081496c30c5e7d085de66cd2d63504e92d830aaebce (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:44:45.556 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 podman[62699]: 2024-03-26 02:44:45.134496421 +0000 UTC m=+0.428990231 container cleanup 114ffb73b2787362f22fc081496c30c5e7d085de66cd2d63504e92d830aaebce (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:44:45.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 bash[62699]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T02:44:45.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 podman[62713]: 2024-03-26 02:44:45.352705494 +0000 UTC m=+0.544735760 container remove 114ffb73b2787362f22fc081496c30c5e7d085de66cd2d63504e92d830aaebce (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:44:45.557 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:44:45.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:45 smithi172 ceph-mon[25688]: Upgrade: Updating node-exporter.b (2/2) 2024-03-26T02:44:45.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:45 smithi172 ceph-mon[25688]: Deploying daemon node-exporter.b on smithi172 2024-03-26T02:44:45.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T02:44:45.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 systemd[1]: Stopped Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:44:45.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 5.048s CPU time. 2024-03-26T02:44:45.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 systemd[1]: Starting Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:44:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:45 smithi164 ceph-mon[24851]: Upgrade: Updating node-exporter.b (2/2) 2024-03-26T02:44:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:45 smithi164 ceph-mon[24851]: Deploying daemon node-exporter.b on smithi172 2024-03-26T02:44:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:45 smithi164 ceph-mon[29512]: Upgrade: Updating node-exporter.b (2/2) 2024-03-26T02:44:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:45 smithi164 ceph-mon[29512]: Deploying daemon node-exporter.b on smithi172 2024-03-26T02:44:46.306 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:45 smithi172 bash[62841]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T02:44:46.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:46 smithi172 ceph-mon[25688]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:44:46.807 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:46 smithi172 bash[62841]: Getting image source signatures 2024-03-26T02:44:46.807 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:46 smithi172 bash[62841]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T02:44:46.807 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:46 smithi172 bash[62841]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T02:44:46.807 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:46 smithi172 bash[62841]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T02:44:47.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:46 smithi164 ceph-mon[24851]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:44:47.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:46 smithi164 ceph-mon[29512]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:44:47.995 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:47 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:47.995 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:47 smithi172 bash[62841]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T02:44:48.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:47 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:48.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:47 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:48.307 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:47 smithi172 bash[62841]: Writing manifest to image destination 2024-03-26T02:44:48.307 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 podman[62841]: 2024-03-26 02:44:48.208789299 +0000 UTC m=+2.307565961 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 podman[62841]: 2024-03-26 02:44:48.315373376 +0000 UTC m=+2.414150028 container create 352ab9d69d2a8cfa784a3d8a3f7b136ca37c227d13f7dddb82317eb352db561c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 podman[62841]: 2024-03-26 02:44:48.542685405 +0000 UTC m=+2.641462054 container init 352ab9d69d2a8cfa784a3d8a3f7b136ca37c227d13f7dddb82317eb352db561c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 podman[62841]: 2024-03-26 02:44:48.545571247 +0000 UTC m=+2.644347897 container start 352ab9d69d2a8cfa784a3d8a3f7b136ca37c227d13f7dddb82317eb352db561c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.549Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.549Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T02:44:48.598 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:44:48.599 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T02:44:48.599 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T02:44:48.599 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T02:44:48.599 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T02:44:48.599 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T02:44:48.599 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T02:44:48.599 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T02:44:48.599 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T02:44:48.600 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.550Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T02:44:48.601 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T02:44:48.602 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b[62890]: ts=2024-03-26T02:44:48.551Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T02:44:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:48 smithi164 ceph-mon[24851]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:48 smithi164 ceph-mon[29512]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:48 smithi172 ceph-mon[25688]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:49.057 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 bash[62841]: 352ab9d69d2a8cfa784a3d8a3f7b136ca37c227d13f7dddb82317eb352db561c 2024-03-26T02:44:49.057 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 02:44:48 smithi172 systemd[1]: Started Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:44:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:49 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:49 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:49 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:49 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:49 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:49 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:49 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:49 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:49 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:50 smithi164 ceph-mon[24851]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:50 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:50 smithi164 ceph-mon[29512]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:50 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:50 smithi172 ceph-mon[25688]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:50 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:44:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:44:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:44:51] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.33.4" 2024-03-26T02:44:52.963 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:52 smithi164 ceph-mon[24851]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:52.963 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:52 smithi164 ceph-mon[29512]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:52 smithi172 ceph-mon[25688]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:54.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[24851]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:54.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[29512]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:54.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:53 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:53 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:53 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:53 smithi172 ceph-mon[25688]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:53 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:53 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.234 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:55 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.235 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:55 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.235 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:55 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.235 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:55 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.235 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:55 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.235 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:55 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.235 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:55 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.235 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:55 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:55 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:55 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:55 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:55 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:56.193 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:56 smithi164 ceph-mon[24851]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:56.194 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:56 smithi164 ceph-mon[29512]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:56.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:56 smithi172 ceph-mon[25688]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:58.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T02:44:58.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:44:58 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:44:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 975 B/s rd, 0 op/s 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:44:58.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:44:58 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:00.965 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:00 smithi172 ceph-mon[25688]: Upgrade: Updating prometheus.a 2024-03-26T02:45:00.965 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:00 smithi172 ceph-mon[25688]: Deploying daemon prometheus.a on smithi172 2024-03-26T02:45:00.965 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:00 smithi172 ceph-mon[25688]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-03-26T02:45:01.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:00 smithi164 ceph-mon[29512]: Upgrade: Updating prometheus.a 2024-03-26T02:45:01.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:00 smithi164 ceph-mon[29512]: Deploying daemon prometheus.a on smithi172 2024-03-26T02:45:01.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:00 smithi164 ceph-mon[29512]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-03-26T02:45:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:00 smithi164 ceph-mon[24851]: Upgrade: Updating prometheus.a 2024-03-26T02:45:01.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:00 smithi164 ceph-mon[24851]: Deploying daemon prometheus.a on smithi172 2024-03-26T02:45:01.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:00 smithi164 ceph-mon[24851]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-03-26T02:45:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:45:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:45:01] "GET /metrics HTTP/1.1" 200 37999 "" "Prometheus/2.33.4" 2024-03-26T02:45:02.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:02 smithi172 ceph-mon[25688]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:03.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:02 smithi164 ceph-mon[24851]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:03.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:02 smithi164 ceph-mon[29512]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:02 smithi172 systemd[1]: Stopping Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:45:03.146 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-03-26T02:45:03.147 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.135Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:45:03.147 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.136Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:45:03.147 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.136Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-03-26T02:45:03.147 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[60851]: ts=2024-03-26T02:45:03.136Z caller=main.go:1066 level=info msg="See you next time!" 2024-03-26T02:45:03.147 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 podman[64206]: 2024-03-26 02:45:03.14550751 +0000 UTC m=+0.116534093 container died 35f9aa6bb3b6c2304d218a0cca91cfc8bb82ac9d8389dd8fcfd879a3377089b9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:03.728 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 podman[64206]: 2024-03-26 02:45:03.53590251 +0000 UTC m=+0.506929100 container cleanup 35f9aa6bb3b6c2304d218a0cca91cfc8bb82ac9d8389dd8fcfd879a3377089b9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:03.728 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 bash[64206]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:45:04.056 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 podman[64220]: 2024-03-26 02:45:03.72846499 +0000 UTC m=+0.584794042 container remove 35f9aa6bb3b6c2304d218a0cca91cfc8bb82ac9d8389dd8fcfd879a3377089b9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:04.056 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T02:45:04.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 systemd[1]: Stopped Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:45:04.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:03 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.257s CPU time. 2024-03-26T02:45:04.502 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 systemd[1]: Starting Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:45:04.502 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 podman[64352]: 2024-03-26 02:45:04.274178269 +0000 UTC m=+0.018919368 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T02:45:04.502 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 podman[64352]: 2024-03-26 02:45:04.396528326 +0000 UTC m=+0.141269427 container create 6e31603f6658434213d279238b5b4940c6bb0c5a09c8e4e7bada10d57998ee1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:05.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:04 smithi164 ceph-mon[24851]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:05.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:04 smithi164 ceph-mon[29512]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:05.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-mon[25688]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 podman[64352]: 2024-03-26 02:45:04.607087786 +0000 UTC m=+0.351828885 container init 6e31603f6658434213d279238b5b4940c6bb0c5a09c8e4e7bada10d57998ee1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 podman[64352]: 2024-03-26 02:45:04.610001877 +0000 UTC m=+0.354742976 container start 6e31603f6658434213d279238b5b4940c6bb0c5a09c8e4e7bada10d57998ee1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.633Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.633Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.633Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi172 (none))" 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.633Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.633Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.635Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:45:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.635Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.636Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.636Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.639Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.639Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.666µs 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.639Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.644Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=6 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.658Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=6 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.669Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=6 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.672Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=6 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.675Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=4 maxSegment=6 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.676Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=5 maxSegment=6 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.676Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=6 maxSegment=6 2024-03-26T02:45:05.058 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.676Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.441µs wal_replay_duration=36.976125ms wbl_replay_duration=467ns total_replay_duration=37.025925ms 2024-03-26T02:45:05.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.679Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:45:05.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.679Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T02:45:05.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.679Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:45:05.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.700Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.109348ms db_storage=824ns remote_storage=1.103µs web_handler=411ns query_engine=604ns scrape=422.215µs scrape_sd=95.379µs notify=10.204µs notify_sd=8.367µs rules=20.269195ms tracing=6.03µs 2024-03-26T02:45:05.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.700Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T02:45:05.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:45:04.700Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T02:45:05.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 bash[64352]: 6e31603f6658434213d279238b5b4940c6bb0c5a09c8e4e7bada10d57998ee1f 2024-03-26T02:45:05.059 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:45:04 smithi172 systemd[1]: Started Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:45:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:07.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:06 smithi164 ceph-mon[24851]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-03-26T02:45:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:06 smithi164 ceph-mon[29512]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-03-26T02:45:07.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:06 smithi172 ceph-mon[25688]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 602 B/s rd, 0 op/s 2024-03-26T02:45:08.080 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:45:08.735 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:45:08.735 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (13m) 15s ago 21m 24.0M - 0.23.0 ba2b418f427c 98dbe984702c 2024-03-26T02:45:08.735 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 error 0s ago 21m - - 2024-03-26T02:45:08.735 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (52s) 15s ago 20m 44.3M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:45:08.736 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (42s) 0s ago 26m 449M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:45:08.736 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (10m) 15s ago 27m 500M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:45:08.736 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (27m) 15s ago 27m 77.1M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:45:08.736 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (26m) 0s ago 26m 57.9M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:45:08.736 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (26m) 15s ago 26m 55.4M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:45:08.736 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (26s) 15s ago 22m 7884k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:45:08.736 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (20s) 0s ago 22m 8568k - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:45:08.736 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (25m) 15s ago 25m 54.9M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:45:08.737 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (25m) 15s ago 25m 56.5M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:45:08.737 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (25m) 15s ago 25m 53.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:45:08.737 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (24m) 15s ago 24m 57.2M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:45:08.737 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (24m) 0s ago 24m 56.2M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:45:08.737 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (23m) 0s ago 23m 52.3M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:45:08.737 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (23m) 0s ago 23m 54.7M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:45:08.738 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (23m) 0s ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:45:08.738 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (4s) 0s ago 22m 48.6M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:45:08.738 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (20m) 15s ago 20m 76.9M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:45:08.738 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (20m) 0s ago 20m 79.4M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:45:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:08 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:08 smithi164 ceph-mon[24851]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:08 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:08 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:09.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:08 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:09.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:08 smithi164 ceph-mon[29512]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:09.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:08 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:09.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:08 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:08 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:08 smithi172 ceph-mon[25688]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:08 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:08 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:09.134 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:45:09.134 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:45:09.134 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:45:09.134 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:45:09.135 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [ 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr" 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "2/2 daemons upgraded", 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading prometheus daemons", 2024-03-26T02:45:09.465 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:45:09.466 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:45:10.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[29512]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[29512]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[29512]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/4140679127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:10.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:10.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:10.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[24851]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[24851]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[24851]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/4140679127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:10.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:10.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:09 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:09 smithi172 ceph-mon[25688]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:09 smithi172 ceph-mon[25688]: from='client.25420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:09 smithi172 ceph-mon[25688]: from='client.25426 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:09 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/4140679127' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:09 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:10.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:09 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:10 smithi164 ceph-mon[24851]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:10 smithi164 ceph-mon[24851]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:10 smithi164 ceph-mon[29512]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:10 smithi164 ceph-mon[29512]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:10 smithi172 ceph-mon[25688]: from='client.25438 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:10 smithi172 ceph-mon[25688]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:45:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:45:12] "GET /metrics HTTP/1.1" 200 37998 "" "Prometheus/2.43.0" 2024-03-26T02:45:12.910 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:12 smithi172 ceph-mon[25688]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:12 smithi164 ceph-mon[24851]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:13.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:12 smithi164 ceph-mon[29512]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:14.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:14.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:14.297 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.298 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:13 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:14.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:13 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:15.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:14 smithi164 ceph-mon[24851]: Upgrade: Updating alertmanager.a 2024-03-26T02:45:15.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:14 smithi164 ceph-mon[24851]: Deploying daemon alertmanager.a on smithi164 2024-03-26T02:45:15.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:14 smithi164 ceph-mon[29512]: Upgrade: Updating alertmanager.a 2024-03-26T02:45:15.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:14 smithi164 ceph-mon[29512]: Deploying daemon alertmanager.a on smithi164 2024-03-26T02:45:15.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:14 smithi172 ceph-mon[25688]: Upgrade: Updating alertmanager.a 2024-03-26T02:45:15.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:14 smithi172 ceph-mon[25688]: Deploying daemon alertmanager.a on smithi164 2024-03-26T02:45:16.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:15 smithi164 ceph-mon[24851]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:16.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:15 smithi164 ceph-mon[29512]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:16.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:15 smithi172 ceph-mon[25688]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:18.103 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:17 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:18.103 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:17 smithi164 ceph-mon[29512]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:18.103 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:17 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:18.103 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:17 smithi164 ceph-mon[24851]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:18.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:17 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:18.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:17 smithi172 ceph-mon[25688]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:18.391 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:18 smithi164 systemd[1]: Stopping Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:45:18.391 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:18 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[59143]: level=info ts=2024-03-26T02:45:18.350Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:45:18.391 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:18 smithi164 podman[78623]: 2024-03-26 02:45:18.390861474 +0000 UTC m=+0.148186366 container died 98dbe984702c222e227e51664578839258e1fb25c97ebdbc6c404612607f38fe (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:19.026 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:18 smithi164 podman[78623]: 2024-03-26 02:45:18.758579641 +0000 UTC m=+0.515904529 container cleanup 98dbe984702c222e227e51664578839258e1fb25c97ebdbc6c404612607f38fe (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:19.026 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:18 smithi164 bash[78623]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:45:19.294 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 podman[78637]: 2024-03-26 02:45:19.026505712 +0000 UTC m=+0.628205369 container remove 98dbe984702c222e227e51664578839258e1fb25c97ebdbc6c404612607f38fe (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:19.294 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 podman[78637]: 2024-03-26 02:45:19.127411047 +0000 UTC m=+0.729110706 volume remove 1ffc7feb0a4e9ee1cb8bda6ce5846783070ab9debae439e6e2885b55b07c51cd 2024-03-26T02:45:19.753 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:45:19.754 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 systemd[1]: Stopped Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:45:19.754 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.565s CPU time. 2024-03-26T02:45:19.754 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 systemd[1]: Starting Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:45:20.035 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 podman[78768]: 2024-03-26 02:45:19.75323417 +0000 UTC m=+0.110136111 volume create e1275d27c34529ba0eb2d0d6bb9f665ac538e449dd8f1c388ad96b06a09e60ee 2024-03-26T02:45:20.035 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 podman[78768]: 2024-03-26 02:45:19.661900174 +0000 UTC m=+0.018802099 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T02:45:20.035 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:19 smithi164 podman[78768]: 2024-03-26 02:45:19.845416708 +0000 UTC m=+0.202318648 container create 763628284e16f991a5be32b28bee39f60aba703190f0bd1eaa148cb07a010f2c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:20.294 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 podman[78768]: 2024-03-26 02:45:20.140286952 +0000 UTC m=+0.497188853 container init 763628284e16f991a5be32b28bee39f60aba703190f0bd1eaa148cb07a010f2c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:20.294 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 podman[78768]: 2024-03-26 02:45:20.142900811 +0000 UTC m=+0.499802715 container start 763628284e16f991a5be32b28bee39f60aba703190f0bd1eaa148cb07a010f2c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:20.154Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:20.155Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:20.156Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.164 port=9094 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:20.157Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:20.181Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:20.181Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:20.184Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:20.184Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 bash[78768]: 763628284e16f991a5be32b28bee39f60aba703190f0bd1eaa148cb07a010f2c 2024-03-26T02:45:20.295 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:20 smithi164 systemd[1]: Started Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:45:21.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:20 smithi172 ceph-mon[25688]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:21.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:20 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:21.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:20 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:21.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:20 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:21.010 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:20 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[29512]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[24851]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:20 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:45:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:45:22] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-03-26T02:45:22.294 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:22.158Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000876161s 2024-03-26T02:45:22.775 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:22 smithi164 ceph-mon[24851]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:22.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:22 smithi164 ceph-mon[29512]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:22.844 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:22 smithi172 ceph-mon[25688]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:45:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:24 smithi164 ceph-mon[24851]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:24 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:24 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:24 smithi164 ceph-mon[29512]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:24 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:24 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:24 smithi172 ceph-mon[25688]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:24 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:24 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:25 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:25 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:26.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:25 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:26.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:25 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:25 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:25 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:26 smithi172 ceph-mon[25688]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:26 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:27.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:26 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:27.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:26 smithi164 ceph-mon[24851]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:27.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:26 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:27.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:26 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:27.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:26 smithi164 ceph-mon[29512]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:27.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:26 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:27.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:26 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:29.042 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:28 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:29.042 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:28 smithi164 ceph-mon[29512]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:29.043 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:28 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:29.043 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:28 smithi164 ceph-mon[24851]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:29.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:28 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:29.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:28 smithi172 ceph-mon[25688]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:30.516 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:45:30.161Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003591984s 2024-03-26T02:45:30.543 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:30.543 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.543 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.544 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.545 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:30 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 972 B/s rd, 0 op/s 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:30.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:45:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:45:31] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-03-26T02:45:32.787 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:32 smithi172 ceph-mon[25688]: Upgrade: Updating grafana.a 2024-03-26T02:45:32.787 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:32 smithi172 ceph-mon[25688]: Deploying daemon grafana.a on smithi172 2024-03-26T02:45:32.787 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:32 smithi172 ceph-mon[25688]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:32 smithi164 ceph-mon[24851]: Upgrade: Updating grafana.a 2024-03-26T02:45:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:32 smithi164 ceph-mon[24851]: Deploying daemon grafana.a on smithi172 2024-03-26T02:45:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:32 smithi164 ceph-mon[24851]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:33.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:32 smithi164 ceph-mon[29512]: Upgrade: Updating grafana.a 2024-03-26T02:45:33.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:32 smithi164 ceph-mon[29512]: Deploying daemon grafana.a on smithi172 2024-03-26T02:45:33.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:32 smithi164 ceph-mon[29512]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:34.548 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 systemd[1]: Starting Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:45:34.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-mon[25688]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:34.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 podman[66316]: 2024-03-26 02:45:34.450718492 +0000 UTC m=+0.018489482 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-26T02:45:34.807 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 podman[66316]: 2024-03-26 02:45:34.579435972 +0000 UTC m=+0.147206950 container create c46cfc9279a0762ac3525fb1317c7572bffc5019261651f6fc439e09b9537971 (image=quay.io/ceph/grafana:9.4.12, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T02:45:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:34 smithi164 ceph-mon[24851]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:35.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:34 smithi164 ceph-mon[29512]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:35.061 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 podman[66316]: 2024-03-26 02:45:34.831991129 +0000 UTC m=+0.399762109 container init c46cfc9279a0762ac3525fb1317c7572bffc5019261651f6fc439e09b9537971 (image=quay.io/ceph/grafana:9.4.12, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T02:45:35.061 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 podman[66316]: 2024-03-26 02:45:34.834796169 +0000 UTC m=+0.402567153 container start c46cfc9279a0762ac3525fb1317c7572bffc5019261651f6fc439e09b9537971 (image=quay.io/ceph/grafana:9.4.12, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T02:45:35.061 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 bash[66316]: c46cfc9279a0762ac3525fb1317c7572bffc5019261651f6fc439e09b9537971 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 systemd[1]: Started Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983704249Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983867399Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983877717Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983880999Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.98388346Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983886296Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983889308Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983891718Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983894103Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983896766Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983899223Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983901536Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T02:45:35.062 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983903935Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T02:45:35.063 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983906603Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T02:45:35.063 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983908921Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T02:45:35.063 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983911203Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T02:45:35.063 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983913455Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T02:45:35.063 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.983915942Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T02:45:35.063 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=settings t=2024-03-26T02:45:34.98391852Z level=info msg="App mode production" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=sqlstore t=2024-03-26T02:45:34.983975849Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=sqlstore t=2024-03-26T02:45:34.983987079Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:34.99517432Z level=info msg="Starting DB migrations" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:34.998588212Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.009891545Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.011023648Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.01181855Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.013073435Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.014143315Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.015186422Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.016217851Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.016772086Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.017273366Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T02:45:35.064 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.018128025Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.019248792Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.019605471Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.020700796Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.02178918Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.022871189Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.023730964Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.02482266Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.025893931Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.026762914Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.027847182Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.028920027Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.029787626Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.030662372Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.031490217Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T02:45:35.065 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.032391231Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.033210713Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.034127274Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.034935631Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.03501504Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.035787082Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.036649003Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.037536871Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.04039125Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.043135236Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.049475124Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.052246093Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.055118873Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T02:45:35.066 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.061459163Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T02:45:35.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.069041242Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T02:45:35.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.071820735Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T02:45:35.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.074730624Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T02:45:35.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.077611036Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T02:45:35.557 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.080424365Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.083216206Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.08612428Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.088994431Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.091891096Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.09465524Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.097653267Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.100150243Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.102565889Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.105335466Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.108055849Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.110495506Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.113127333Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T02:45:35.558 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.115926393Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.118683662Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.121349389Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.121452763Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.124114063Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.124204443Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.126632587Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.12938613Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.132088399Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.134730396Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.137499203Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.139919607Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.14266394Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T02:45:35.559 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.145341459Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T02:45:35.560 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.146160689Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T02:45:35.560 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.14700287Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T02:45:35.560 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.147885435Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T02:45:35.560 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.148739111Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T02:45:35.560 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.1526937Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T02:45:35.561 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.153882427Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T02:45:35.561 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.155064291Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T02:45:35.561 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.155868971Z level=info msg="Executing migration" id="add share column" 2024-03-26T02:45:35.561 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.157104945Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T02:45:35.561 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.157958024Z level=info msg="Executing migration" id="create file table" 2024-03-26T02:45:35.561 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.158744753Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T02:45:35.561 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.159597719Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T02:45:35.561 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.160460383Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.161318019Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.162160325Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.162938176Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.164068801Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.164952508Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.165844194Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.167095375Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.167972334Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.168826173Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.169668483Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.170522868Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.17135522Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.172623691Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T02:45:35.562 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.173894485Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.174770086Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.188088652Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.188938372Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.189796479Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.19422516Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.195159557Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.195953566Z level=info msg="Executing migration" id="create folder table" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.19683249Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.199674701Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T02:45:35.563 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.202390757Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T02:45:35.564 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.205207796Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T02:45:35.564 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=migrator t=2024-03-26T02:45:35.207968046Z level=info msg="migrations completed" performed=105 skipped=376 duration=209.411699ms 2024-03-26T02:45:35.564 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=sqlstore t=2024-03-26T02:45:35.208321041Z level=info msg="Created default organization" 2024-03-26T02:45:35.564 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=plugin.finder t=2024-03-26T02:45:35.241559288Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-26T02:45:35.564 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=secrets t=2024-03-26T02:45:35.241668758Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T02:45:35.564 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=query_data t=2024-03-26T02:45:35.245209397Z level=info msg="Query Service initialization" 2024-03-26T02:45:35.564 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=live.push_http t=2024-03-26T02:45:35.247772403Z level=info msg="Live Push Gateway initialization" 2024-03-26T02:45:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-mon[25688]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-26T02:45:36.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=infra.usagestats.collector t=2024-03-26T02:45:35.891933358Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T02:45:36.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=provisioning.datasources t=2024-03-26T02:45:35.895101662Z level=info msg="deleted correlations based on configuration" ds_name=Dashboard1 2024-03-26T02:45:36.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=provisioning.datasources t=2024-03-26T02:45:35.895203459Z level=info msg="deleted datasource based on configuration" name=Dashboard1 2024-03-26T02:45:36.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=provisioning.datasources t=2024-03-26T02:45:35.895404184Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T02:45:36.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=provisioning.datasources t=2024-03-26T02:45:35.907801646Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=provisioning.alerting t=2024-03-26T02:45:35.918437205Z level=info msg="starting to provision alerting" 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=provisioning.alerting t=2024-03-26T02:45:35.918448203Z level=info msg="finished to provision alerting" 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=grafanaStorageLogger t=2024-03-26T02:45:35.918690304Z level=info msg="storage starting" 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=ngalert.state.manager t=2024-03-26T02:45:35.918726271Z level=info msg="Warming state cache for startup" 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=http.server t=2024-03-26T02:45:35.921067882Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=ngalert.state.manager t=2024-03-26T02:45:35.928131957Z level=info msg="State cache has been initialized" states=0 duration=9.40463ms 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=ticker t=2024-03-26T02:45:35.928185676Z level=info msg=starting first_tick=2024-03-26T02:45:40Z 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=ngalert.multiorg.alertmanager t=2024-03-26T02:45:35.928192665Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=sqlstore.transactions t=2024-03-26T02:45:35.939963299Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T02:45:36.058 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:45:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=sqlstore.transactions t=2024-03-26T02:45:35.951020926Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T02:45:36.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:36.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:36.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:36.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:36.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[24851]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-26T02:45:36.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:36.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:36.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:36.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:36.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:35 smithi164 ceph-mon[29512]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-26T02:45:38.901 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:38 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:38.901 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:38 smithi172 ceph-mon[25688]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:38 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:38 smithi164 ceph-mon[29512]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:38 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:38 smithi164 ceph-mon[24851]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:39.825 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (20s) 15s ago 22m 14.6M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (6s) 1s ago 21m 49.0M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:45:40.481 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (84s) 15s ago 21m 44.5M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (74s) 1s ago 26m 449M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (11m) 15s ago 28m 503M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (28m) 15s ago 28m 77.9M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (27m) 1s ago 27m 56.8M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (27m) 15s ago 27m 55.5M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (58s) 15s ago 22m 8682k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (52s) 1s ago 22m 16.5M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (26m) 15s ago 26m 54.9M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (26m) 15s ago 26m 56.5M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:45:40.482 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (25m) 15s ago 25m 53.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (25m) 15s ago 25m 57.2M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (24m) 1s ago 24m 56.2M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (24m) 1s ago 24m 52.3M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (24m) 1s ago 24m 54.7M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (23m) 1s ago 23m 55.2M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (36s) 1s ago 23m 51.6M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (21m) 15s ago 21m 77.0M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:45:40.483 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (21m) 1s ago 21m 79.5M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:45:40.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:40 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:40.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:40 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:40.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:40 smithi164 ceph-mon[24851]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:40.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:40 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:40.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:40 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:40.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:40 smithi164 ceph-mon[29512]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:40.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:40 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:40.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:40 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:40.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:40 smithi172 ceph-mon[25688]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:45:40.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:45:40.898 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [ 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr" 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "2/2 daemons upgraded", 2024-03-26T02:45:41.288 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading grafana daemons", 2024-03-26T02:45:41.289 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:45:41.289 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:45:41.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[24851]: from='client.25444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[24851]: from='client.25450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:41.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:41.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[24851]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3285622431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:41.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[29512]: from='client.25444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[29512]: from='client.25450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:41.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:41.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[29512]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:41 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3285622431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:41.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:41 smithi172 ceph-mon[25688]: from='client.25444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:41 smithi172 ceph-mon[25688]: from='client.25450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:41 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:41.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:41 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:41.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:41 smithi172 ceph-mon[25688]: from='client.15552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:41.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:41 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3285622431' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:42.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:45:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:45:42] "GET /metrics HTTP/1.1" 200 37995 "" "Prometheus/2.43.0" 2024-03-26T02:45:42.402 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:42 smithi172 ceph-mon[25688]: from='client.25468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:42.402 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:42 smithi172 ceph-mon[25688]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:42.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:42 smithi164 ceph-mon[29512]: from='client.25468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:42.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:42 smithi164 ceph-mon[29512]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:42.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:42 smithi164 ceph-mon[24851]: from='client.25468 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:45:42.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:42 smithi164 ceph-mon[24851]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:45.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:45.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T02:45:45.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:45:45.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:45.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T02:45:45.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:45:45.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:45:45.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:45:45.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:44 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:45:45.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:45:45.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:44 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[24851]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:45:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[24851]: Upgrade: Finalizing container_image settings 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[24851]: Upgrade: Complete! 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[24851]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[24851]: Cluster is now healthy 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[29512]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[29512]: Upgrade: Finalizing container_image settings 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[29512]: Upgrade: Complete! 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[29512]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[29512]: Cluster is now healthy 2024-03-26T02:45:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:45 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:45 smithi172 ceph-mon[25688]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:45:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:45 smithi172 ceph-mon[25688]: Upgrade: Finalizing container_image settings 2024-03-26T02:45:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:45 smithi172 ceph-mon[25688]: Upgrade: Complete! 2024-03-26T02:45:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:45 smithi172 ceph-mon[25688]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-03-26T02:45:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:45 smithi172 ceph-mon[25688]: Cluster is now healthy 2024-03-26T02:45:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:45 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:47.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:46 smithi164 ceph-mon[24851]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T02:45:47.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:46 smithi164 ceph-mon[29512]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T02:45:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:46 smithi172 ceph-mon[25688]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T02:45:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:48 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:48 smithi164 ceph-mon[24851]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-26T02:45:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:48 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:48 smithi164 ceph-mon[29512]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-26T02:45:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:48 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:49.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:48 smithi172 ceph-mon[25688]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-26T02:45:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:50 smithi164 ceph-mon[24851]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-26T02:45:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:50 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:50 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:50 smithi164 ceph-mon[29512]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-26T02:45:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:50 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:50 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:50 smithi172 ceph-mon[25688]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-03-26T02:45:51.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:50 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:45:51.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:50 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:45:52.297 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:45:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:45:52] "GET /metrics HTTP/1.1" 200 37992 "" "Prometheus/2.43.0" 2024-03-26T02:45:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:52 smithi164 ceph-mon[24851]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:53.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:52 smithi164 ceph-mon[29512]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:52 smithi172 ceph-mon[25688]: pgmap v64: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:55.017 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:54 smithi172 ceph-mon[25688]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:54 smithi164 ceph-mon[24851]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:55.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:54 smithi164 ceph-mon[29512]: pgmap v65: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 902 B/s rd, 0 op/s 2024-03-26T02:45:57.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:56 smithi172 ceph-mon[25688]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:45:57.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:56 smithi164 ceph-mon[24851]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:45:57.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:56 smithi164 ceph-mon[29512]: pgmap v66: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:45:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:58 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:45:58 smithi172 ceph-mon[25688]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:59.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:58 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:59.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:45:58 smithi164 ceph-mon[24851]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:45:59.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:58 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:45:59.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:45:58 smithi164 ceph-mon[29512]: pgmap v67: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:01.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:00 smithi172 ceph-mon[25688]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:46:01.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:00 smithi164 ceph-mon[24851]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:46:01.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:00 smithi164 ceph-mon[29512]: pgmap v68: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:46:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:46:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:46:02] "GET /metrics HTTP/1.1" 200 37992 "" "Prometheus/2.43.0" 2024-03-26T02:46:03.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:02 smithi164 ceph-mon[24851]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:03.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:02 smithi164 ceph-mon[29512]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:02 smithi172 ceph-mon[25688]: pgmap v69: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:04.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:04 smithi172 ceph-mon[25688]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:46:05.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:04 smithi164 ceph-mon[24851]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:46:05.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:04 smithi164 ceph-mon[29512]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:46:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:05 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:06.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:06 smithi164 ceph-mon[24851]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:06.865 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:06 smithi164 ceph-mon[29512]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:07.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:06 smithi172 ceph-mon[25688]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:08 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:08 smithi172 ceph-mon[25688]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:09.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:08 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:09.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:08 smithi164 ceph-mon[24851]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:09.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:08 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:09.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:08 smithi164 ceph-mon[29512]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:10 smithi172 ceph-mon[25688]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:11.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:10 smithi164 ceph-mon[24851]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:11.097 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:10 smithi164 ceph-mon[29512]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:46:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:46:11] "GET /metrics HTTP/1.1" 200 37994 "" "Prometheus/2.43.0" 2024-03-26T02:46:12.447 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-26T02:46:13.009 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:12 smithi164 ceph-mon[24851]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:13.009 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:12 smithi164 ceph-mon[24851]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:13.009 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:12 smithi164 ceph-mon[29512]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:13.009 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:12 smithi164 ceph-mon[29512]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:13.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:12 smithi172 ceph-mon[25688]: from='client.25474 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:13.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:12 smithi172 ceph-mon[25688]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:14.467 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:46:14.901 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:14 smithi164 ceph-mon[24851]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:14.901 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:14 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1894322202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:14.901 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:14 smithi164 ceph-mon[29512]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:14.901 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:14 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1894322202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:14 smithi172 ceph-mon[25688]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:14 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1894322202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:15.140 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-26T02:46:17.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:16 smithi164 ceph-mon[24851]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:17.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:16 smithi164 ceph-mon[29512]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:17.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:16 smithi172 ceph-mon[25688]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:17.189 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)" 2024-03-26T02:46:17.954 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:17 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2001200647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:17.955 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:17 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2001200647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:17.981 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 2'"'"'' 2024-03-26T02:46:18.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:17 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2001200647' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:18 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:18 smithi164 ceph-mon[24851]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:18 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:18 smithi164 ceph-mon[29512]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:19.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:18 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:19.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:18 smithi172 ceph-mon[25688]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:20.074 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:46:20.712 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 2'"'"'' 2024-03-26T02:46:21.022 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:20 smithi164 ceph-mon[24851]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:21.022 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:20 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:21.022 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:20 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3068392916' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:21.023 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:20 smithi164 ceph-mon[29512]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:21.023 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:20 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:21.023 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:20 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3068392916' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:20 smithi172 ceph-mon[25688]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:20 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:20 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/3068392916' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:21 smithi164 ceph-mon[24851]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:22.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:21 smithi164 ceph-mon[29512]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:22.208 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:21 smithi172 ceph-mon[25688]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:46:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:46:21] "GET /metrics HTTP/1.1" 200 38003 "" "Prometheus/2.43.0" 2024-03-26T02:46:23.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:22 smithi164 ceph-mon[24851]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:23.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:22 smithi164 ceph-mon[29512]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:23.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:22 smithi172 ceph-mon[25688]: from='client.15594 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:24.773 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:46:24.905 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:24 smithi164 ceph-mon[24851]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:24.905 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:24 smithi164 ceph-mon[29512]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:25.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:24 smithi172 ceph-mon[25688]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:25.453 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade status' 2024-03-26T02:46:26.927 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:26 smithi164 ceph-mon[24851]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:26.928 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:26 smithi164 ceph-mon[29512]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:26 smithi172 ceph-mon[25688]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": null, 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": false, 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "", 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": null, 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:46:27.376 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:46:27.976 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:46:28.992 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:28 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:28.992 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:28 smithi164 ceph-mon[24851]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:28.992 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:28 smithi164 ceph-mon[24851]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:28.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:28 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:28.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:28 smithi164 ceph-mon[29512]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:28.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:28 smithi164 ceph-mon[29512]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:29.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:28 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:29.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:28 smithi172 ceph-mon[25688]: from='client.15600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:29.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:28 smithi172 ceph-mon[25688]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:30.129 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:46:30.751 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.x | awk '"'"'{print $2}'"'"')' 2024-03-26T02:46:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:30 smithi164 ceph-mon[29512]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:30 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2856037936' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:46:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:30 smithi164 ceph-mon[24851]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:31.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:30 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2856037936' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:46:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:30 smithi172 ceph-mon[25688]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:30 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/2856037936' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:46:32.140 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:31 smithi164 ceph-mon[24851]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:32.140 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:46:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:46:31] "GET /metrics HTTP/1.1" 200 38003 "" "Prometheus/2.43.0" 2024-03-26T02:46:32.140 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:31 smithi164 ceph-mon[29512]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:32.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:31 smithi172 ceph-mon[25688]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:32 smithi164 ceph-mon[24851]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:33.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:32 smithi164 ceph-mon[29512]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:32 smithi172 ceph-mon[25688]: from='client.15612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:35.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:34 smithi164 ceph-mon[29512]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mon", "hosts": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:35.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:34 smithi164 ceph-mon[29512]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:35.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:34 smithi164 ceph-mon[24851]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mon", "hosts": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:35.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:34 smithi164 ceph-mon[24851]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:34 smithi172 ceph-mon[25688]: from='client.15618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mon", "hosts": "smithi172", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:34 smithi172 ceph-mon[25688]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:35.339 INFO:teuthology.orchestra.run.smithi164.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:46:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:35 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:35.937 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:35.937 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:35.937 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:35 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:35.960 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-26T02:46:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:35 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:36.896 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[24851]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[24851]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[29512]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[29512]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:36.897 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:36 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:36 smithi172 ceph-mon[25688]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:46:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:36 smithi172 ceph-mon[25688]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:36 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:36 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:36 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:37.701 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:37 smithi164 ceph-mon[29512]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:46:37.702 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:37 smithi164 ceph-mon[24851]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:46:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:37 smithi172 ceph-mon[25688]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:46:38.202 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:46:38.773 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:38 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:38.773 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:38 smithi164 ceph-mon[24851]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:38.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:38 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:38.773 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:38 smithi164 ceph-mon[29512]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (79s) 74s ago 23m 14.6M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (64s) 59s ago 22m 49.0M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (2m) 74s ago 22m 44.5M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (2m) 59s ago 27m 449M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (12m) 74s ago 29m 503M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (29m) 74s ago 29m 77.9M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (27m) 59s ago 27m 56.8M 2048M 17.2.0 e1d6a67b021e e46e34873348 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (28m) 74s ago 28m 55.5M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (117s) 74s ago 23m 8682k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (110s) 59s ago 23m 16.5M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:46:38.873 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (27m) 74s ago 27m 54.9M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (27m) 74s ago 27m 56.5M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (26m) 74s ago 26m 53.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (26m) 74s ago 26m 57.2M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (25m) 59s ago 25m 56.2M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (25m) 59s ago 25m 52.3M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (25m) 59s ago 25m 54.7M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (24m) 59s ago 24m 55.2M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (94s) 59s ago 24m 51.6M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (22m) 74s ago 22m 77.0M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:46:38.874 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (22m) 59s ago 22m 79.5M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:46:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:38 smithi172 ceph-mon[25688]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:38 smithi172 ceph-mon[25688]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:39.280 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:46:39.280 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:46:39.280 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-03-26T02:46:39.280 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:46:39.280 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:46:39.280 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:46:39.280 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:46:39.280 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:46:39.281 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:46:39.614 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:46:39.617 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:46:39.617 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:46:39.617 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi172", 2024-03-26T02:46:39.617 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:46:39.618 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "0/1 daemons upgraded", 2024-03-26T02:46:39.618 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading mon daemons", 2024-03-26T02:46:39.618 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:46:39.618 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:46:39.861 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:39.862 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:39.862 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:39.862 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:39.862 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:39.862 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:39.862 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:46:39.862 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-26T02:46:39.862 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:39 smithi172 ceph-mon[25688]: from='client.? 172.21.15.164:0/1577313963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/1577313963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='client.25528 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='client.25534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:46:40.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-03-26T02:46:40.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:39 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/1577313963' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[24851]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[24851]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[24851]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all mgr 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[24851]: Upgrade: It appears safe to stop mon.b 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[24851]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[24851]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[29512]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[29512]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[29512]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all mgr 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[29512]: Upgrade: It appears safe to stop mon.b 2024-03-26T02:46:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[29512]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:40 smithi164 ceph-mon[29512]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:40 smithi172 ceph-mon[25688]: from='client.25540 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:40 smithi172 ceph-mon[25688]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:46:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:40 smithi172 ceph-mon[25688]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:46:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:40 smithi172 ceph-mon[25688]: Upgrade: Setting container_image for all mgr 2024-03-26T02:46:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:40 smithi172 ceph-mon[25688]: Upgrade: It appears safe to stop mon.b 2024-03-26T02:46:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:40 smithi172 ceph-mon[25688]: from='client.15648 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:46:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:40 smithi172 ceph-mon[25688]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:41 smithi172 ceph-mon[25688]: Upgrade: Updating mon.b 2024-03-26T02:46:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:41 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:41 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:46:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:41 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:46:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:41 smithi172 ceph-mon[25688]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:41 smithi172 ceph-mon[25688]: Deploying daemon mon.b on smithi172 2024-03-26T02:46:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:41 smithi172 ceph-mon[25688]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:42.196 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:46:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:46:41] "GET /metrics HTTP/1.1" 200 38003 "" "Prometheus/2.43.0" 2024-03-26T02:46:42.197 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[24851]: Upgrade: Updating mon.b 2024-03-26T02:46:42.197 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:42.197 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:46:42.197 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:46:42.197 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:42.197 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[24851]: Deploying daemon mon.b on smithi172 2024-03-26T02:46:42.197 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[24851]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:42.197 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[29512]: Upgrade: Updating mon.b 2024-03-26T02:46:42.198 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:42.198 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:46:42.198 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:46:42.198 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:42.198 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[29512]: Deploying daemon mon.b on smithi172 2024-03-26T02:46:42.198 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:41 smithi164 ceph-mon[29512]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:42.983 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:42 smithi172 systemd[1]: Stopping Ceph mon.b for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:46:43.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b[25666]: 2024-03-26T02:46:42.980+0000 7f923ab5f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:46:43.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b[25666]: 2024-03-26T02:46:42.980+0000 7f923ab5f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T02:46:43.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:43 smithi172 podman[67772]: 2024-03-26 02:46:43.104220168 +0000 UTC m=+0.239584996 container died e46e348733488fd7ebfda0d71a141cedb7ac22a65ed237d698757caeb49ded57 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, GIT_BRANCH=HEAD, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD) 2024-03-26T02:46:43.733 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:43 smithi172 podman[67772]: 2024-03-26 02:46:43.456157975 +0000 UTC m=+0.591522807 container cleanup e46e348733488fd7ebfda0d71a141cedb7ac22a65ed237d698757caeb49ded57 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, release=754, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.openshift.expose-services=, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, version=8) 2024-03-26T02:46:43.733 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:43 smithi172 bash[67772]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b 2024-03-26T02:46:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:43 smithi172 podman[67786]: 2024-03-26 02:46:43.732721479 +0000 UTC m=+0.631552117 container remove e46e348733488fd7ebfda0d71a141cedb7ac22a65ed237d698757caeb49ded57 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_BRANCH=HEAD) 2024-03-26T02:46:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:43 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.b.service: Deactivated successfully. 2024-03-26T02:46:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:43 smithi172 systemd[1]: Stopped Ceph mon.b for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:46:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:43 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.b.service: Consumed 17.185s CPU time. 2024-03-26T02:46:44.326 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 systemd[1]: Starting Ceph mon.b for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:46:44.326 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 podman[67918]: 2024-03-26 02:46:44.183737675 +0000 UTC m=+0.018420843 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:46:44.576 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 podman[67918]: 2024-03-26 02:46:44.325678782 +0000 UTC m=+0.160361934 container create 5de9fbe3024c5677f80b4d92f9c3b0c9d976e111aed54f3cac40ddc01064dfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T02:46:44.576 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 podman[67918]: 2024-03-26 02:46:44.528197373 +0000 UTC m=+0.362880526 container init 5de9fbe3024c5677f80b4d92f9c3b0c9d976e111aed54f3cac40ddc01064dfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T02:46:44.576 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 podman[67918]: 2024-03-26 02:46:44.531227932 +0000 UTC m=+0.365911089 container start 5de9fbe3024c5677f80b4d92f9c3b0c9d976e111aed54f3cac40ddc01064dfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:46:44.576 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:46:44.576 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev), process ceph-mon, pid 2 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: pidfile_write: ignore empty --pid-file 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: load: jerasure load: lrc 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: RocksDB version: 7.9.2 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Git sha 0 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Compile date 2024-03-25 23:25:11 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: DB SUMMARY 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: DB Session ID: ZUWXJ4U1WAN95I0UBXJ9 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: MANIFEST file: MANIFEST-000009 size: 2828 Bytes 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000054.sst 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000052.log size: 2623785 ; 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:46:44.577 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.env: 0x560dd46c5c60 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.info_log: 0x560dd6c59ae0 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.statistics: (nil) 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.use_fsync: 0 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:46:44.578 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.db_log_dir: 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.wal_dir: 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:46:44.579 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.write_buffer_manager: 0x560dd6c2a820 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:46:44.580 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:46:44.581 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.unordered_write: 0 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.row_cache: None 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.wal_filter: None 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.wal_compression: 0 2024-03-26T02:46:44.582 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:46:44.583 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_open_files: -1 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:46:44.584 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Compression algorithms supported: 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: kZSTD supported: 0 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.merge_operator: 2024-03-26T02:46:44.585 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_filter: None 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560dd6c59cc0) 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: index_type: 0 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: data_block_index_type: 0 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: index_shortening: 1 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: checksum: 4 2024-03-26T02:46:44.586 INFO:journalctl@ceph.mon.b.smithi172.stdout: no_block_cache: 0 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_cache: 0x560dd6c5d090 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_cache_options: 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: capacity : 536870912 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: num_shard_bits : 4 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: strict_capacity_limit : 0 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_cache_compressed: (nil) 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: persistent_cache: (nil) 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_size: 4096 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_size_deviation: 10 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_restart_interval: 16 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: index_block_restart_interval: 1 2024-03-26T02:46:44.587 INFO:journalctl@ceph.mon.b.smithi172.stdout: metadata_block_size: 4096 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: partition_filters: 0 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: use_delta_encoding: 1 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: filter_policy: bloomfilter 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: whole_key_filtering: 1 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: verify_compression: 0 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: format_version: 5 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: enable_index_compression: 1 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: block_align: 0 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: max_auto_readahead_size: 262144 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: prepopulate_block_cache: 0 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: initial_auto_readahead_size: 8192 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:46:44.588 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression: NoCompression 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.num_levels: 7 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:46:44.589 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:46:44.590 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:46:44.591 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:46:44.592 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.ttl: 2592000 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:46:44.593 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 54.sst 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T02:46:44.594 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 56, last_sequence is 28744, log_number is 52,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 52 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2378f1f5-66ed-4b09-9f01-19ad183ae746 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421204561580, "job": 1, "event": "recovery_started", "wal_files": [52]} 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #52 mode 2 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421204570624, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 57, "file_size": 1616901, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 28749, "largest_seqno": 30304, "table_properties": {"data_size": 1610755, "index_size": 3569, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1541, "raw_key_size": 14357, "raw_average_key_size": 24, "raw_value_size": 1598104, "raw_average_value_size": 2676, "num_data_blocks": 161, "num_entries": 597, "num_filter_entries": 597, "num_deletions": 7, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711421204, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2378f1f5-66ed-4b09-9f01-19ad183ae746", "db_session_id": "ZUWXJ4U1WAN95I0UBXJ9", "orig_file_number": 57, "seqno_to_time_mapping": "N/A"}} 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421204570760, "job": 1, "event": "recovery_finished"} 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/version_set.cc:5047] Creating manifest 59 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000052.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x560dd6d4c000 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: DB pointer 0x560dd6d36000 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T02:46:44.595 INFO:journalctl@ceph.mon.b.smithi172.stdout: ** DB Stats ** 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: L0 1/0 1.54 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 404.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: L6 1/0 8.89 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Sum 2/0 10.43 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 404.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-26T02:46:44.596 INFO:journalctl@ceph.mon.b.smithi172.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 404.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: ** Compaction Stats [default] ** 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 404.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:46:44.597 INFO:journalctl@ceph.mon.b.smithi172.stdout: Cumulative compaction: 0.00 GB write, 115.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: Interval compaction: 0.00 GB write, 115.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: Block cache BinnedLRUCache@0x560dd6c5d090#2 capacity: 512.00 MB usage: 5.86 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.08 KB,0.000777841%) Misc(1,0.00 KB,0%) 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: starting mon.b rank 2 at public addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] at bind addrs [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: mon.b@-1(???) e3 preinit fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: mon.b@-1(???).mds e1 new map 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: mon.b@-1(???).mds e1 print_map 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: e1 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:46:44.598 INFO:journalctl@ceph.mon.b.smithi172.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T02:46:44.599 INFO:journalctl@ceph.mon.b.smithi172.stdout: legacy client fscid: -1 2024-03-26T02:46:44.599 INFO:journalctl@ceph.mon.b.smithi172.stdout: 2024-03-26T02:46:44.599 INFO:journalctl@ceph.mon.b.smithi172.stdout: No filesystems configured 2024-03-26T02:46:44.599 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: mon.b@-1(???).osd e111 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-26T02:46:44.599 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: mon.b@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:46:44.599 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: mon.b@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:46:44.599 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: mon.b@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:46:44.599 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 ceph-mon[67937]: mon.b@-1(???).paxosservice(auth 1..26) refresh upgraded, format 0 -> 3 2024-03-26T02:46:45.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 bash[67918]: 5de9fbe3024c5677f80b4d92f9c3b0c9d976e111aed54f3cac40ddc01064dfbb 2024-03-26T02:46:45.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:44 smithi172 systemd[1]: Started Ceph mon.b for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:46:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: mon.b calling monitor election 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: mon.a calling monitor election 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: monmap e3: 3 mons at {a=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0]} 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: fsmap 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: mgrmap e40: y(active, since 2m), standbys: x 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: overall HEALTH_OK 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: mon.b calling monitor election 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: mon.a calling monitor election 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: monmap e3: 3 mons at {a=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0]} 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: fsmap 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: mgrmap e40: y(active, since 2m), standbys: x 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: overall HEALTH_OK 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:46.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:45 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: mon.b calling monitor election 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: mon.a calling monitor election 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: monmap e3: 3 mons at {a=[v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0],b=[v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0],c=[v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0]} 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: fsmap 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: mgrmap e40: y(active, since 2m), standbys: x 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: overall HEALTH_OK 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:45 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:47.030 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:46 smithi172 ceph-mon[67937]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:47.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:46 smithi164 ceph-mon[29512]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:47.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:46 smithi164 ceph-mon[24851]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:49.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:48 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:49.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:48 smithi164 ceph-mon[24851]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:49.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:48 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:49.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:48 smithi164 ceph-mon[29512]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:49.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:48 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:49.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:48 smithi172 ceph-mon[67937]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:50.020 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:46:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:46:49.581+0000 7f6a15c7c700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-26T02:46:50.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:50 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:50.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:50 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:50.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:50 smithi164 ceph-mon[24851]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:50.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:50 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:50.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:50 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:50.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:50 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:50.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:50 smithi164 ceph-mon[29512]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:50.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:50 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:50.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:50 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:50.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:50 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:50.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:50 smithi172 ceph-mon[67937]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:50.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:50 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:46:51.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:51 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:51.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:51 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:51.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:51 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:51.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:51 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:51.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:51 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:51.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:51 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:52.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:52 smithi164 ceph-mon[29512]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:52.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:52 smithi164 ceph-mon[24851]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:52.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:46:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:46:51] "GET /metrics HTTP/1.1" 200 38013 "" "Prometheus/2.43.0" 2024-03-26T02:46:52.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:52 smithi172 ceph-mon[67937]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi172 2024-03-26T02:46:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all crash 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all mds 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all nfs 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all node-exporter 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Detected new or changed devices on smithi172 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all crash 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all mds 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.799 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all nfs 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.800 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all node-exporter 2024-03-26T02:46:54.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all prometheus 2024-03-26T02:46:54.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all alertmanager 2024-03-26T02:46:54.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all grafana 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all loki 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all promtail 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Finalizing container_image settings 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:46:54.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:46:54.803 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: Upgrade: Complete! 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:46:54.804 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:46:54.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.805 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all prometheus 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all alertmanager 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all grafana 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.806 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all loki 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all promtail 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Finalizing container_image settings 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:46:54.807 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:46:54.808 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: Upgrade: Complete! 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.809 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.810 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.810 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.810 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:54 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Detected new or changed devices on smithi172 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all crash 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mds 2024-03-26T02:46:54.811 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nfs 2024-03-26T02:46:54.812 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all node-exporter 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all prometheus 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all alertmanager 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all grafana 2024-03-26T02:46:54.813 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all loki 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all promtail 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Finalizing container_image settings 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:46:54.814 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:46:54.815 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: Upgrade: Complete! 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:46:54.816 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:54 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:56 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:56 smithi164 ceph-mon[24851]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:56 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:56 smithi164 ceph-mon[29512]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:56 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:46:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:56 smithi172 ceph-mon[67937]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:46:59.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:58 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:59.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:58 smithi164 ceph-mon[29512]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:59.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:58 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:59.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:58 smithi164 ceph-mon[24851]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:46:59.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:58 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:46:59.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:58 smithi172 ceph-mon[67937]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:00.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:46:59 smithi164 ceph-mon[24851]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:00.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:46:59 smithi164 ceph-mon[29512]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:00.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:46:59 smithi172 ceph-mon[67937]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:47:02] "GET /metrics HTTP/1.1" 200 38013 "" "Prometheus/2.43.0" 2024-03-26T02:47:03.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:02 smithi164 ceph-mon[24851]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:02 smithi164 ceph-mon[29512]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:02 smithi172 ceph-mon[67937]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:05.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:04 smithi164 ceph-mon[24851]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:05.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:04 smithi164 ceph-mon[29512]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:05.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:04 smithi172 ceph-mon[67937]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:05 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:05 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:05 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:06.927 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:06 smithi164 ceph-mon[24851]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:06.927 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:06 smithi164 ceph-mon[29512]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:06 smithi172 ceph-mon[67937]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:09.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:08 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:09.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:08 smithi164 ceph-mon[24851]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:09.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:08 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:09.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:08 smithi164 ceph-mon[29512]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:09.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:08 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:08 smithi172 ceph-mon[67937]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:10.293 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:09 smithi164 ceph-mon[24851]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:10.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:09 smithi164 ceph-mon[29512]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:10.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:09 smithi172 ceph-mon[67937]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:10.593 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:47:11.130 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:10 smithi164 ceph-mon[24851]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:11.131 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:10 smithi164 ceph-mon[29512]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:11.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:10 smithi172 ceph-mon[67937]: from='client.15654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:12.130 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:11 smithi164 ceph-mon[29512]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:12.130 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:47:12] "GET /metrics HTTP/1.1" 200 38015 "" "Prometheus/2.43.0" 2024-03-26T02:47:12.131 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:11 smithi164 ceph-mon[24851]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:12.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:11 smithi172 ceph-mon[67937]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:12.585 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:47:12.585 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (112s) 107s ago 23m 14.6M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (98s) 23s ago 23m 75.3M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (2m) 107s ago 22m 44.5M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (2m) 23s ago 28m 451M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (13m) 107s ago 29m 503M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (29m) 107s ago 29m 77.9M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (28s) 23s ago 28m 21.6M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (28m) 107s ago 28m 55.5M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (2m) 107s ago 24m 8682k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (2m) 23s ago 24m 16.3M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (27m) 107s ago 27m 54.9M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (27m) 107s ago 27m 56.5M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (27m) 107s ago 27m 53.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:47:12.586 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (26m) 107s ago 26m 57.2M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:47:12.587 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (26m) 23s ago 26m 56.2M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:47:12.587 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (26m) 23s ago 26m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:47:12.587 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (25m) 23s ago 25m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:47:12.587 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (25m) 23s ago 25m 55.5M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:47:12.587 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (2m) 23s ago 24m 65.6M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:47:12.587 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (23m) 107s ago 23m 77.0M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:47:12.587 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (23m) 23s ago 23m 79.8M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:47:13.275 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:12 smithi164 ceph-mon[24851]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:13.276 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:12 smithi164 ceph-mon[29512]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:13.303 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 2'"'"'' 2024-03-26T02:47:13.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:12 smithi172 ceph-mon[67937]: from='client.25564 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:14.845 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:14 smithi164 ceph-mon[24851]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:14.846 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:14 smithi164 ceph-mon[29512]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:14 smithi172 ceph-mon[67937]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:15.362 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:47:15.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:15 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/2691416913' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:15.699 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:15 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/2691416913' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:16.025 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade status' 2024-03-26T02:47:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:15 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2691416913' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:16.934 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:16 smithi164 ceph-mon[24851]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:16.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:16 smithi164 ceph-mon[29512]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:17.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:16 smithi172 ceph-mon[67937]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:18.026 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:47:18.026 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": null, 2024-03-26T02:47:18.027 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": false, 2024-03-26T02:47:18.027 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "", 2024-03-26T02:47:18.027 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:47:18.027 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": null, 2024-03-26T02:47:18.027 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:47:18.027 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:47:18.027 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:47:18.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:17 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:18.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:17 smithi164 ceph-mon[29512]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:18.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:17 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:18.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:17 smithi164 ceph-mon[24851]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:18.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:17 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:18.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:17 smithi172 ceph-mon[67937]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:18.570 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:47:19.922 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:19 smithi164 ceph-mon[24851]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:19.923 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:19 smithi164 ceph-mon[29512]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:19 smithi172 ceph-mon[67937]: from='client.15672 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:20.723 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:47:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:20 smithi164 ceph-mon[24851]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:20 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:20 smithi164 ceph-mon[29512]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:20 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:20 smithi172 ceph-mon[67937]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:20 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:21.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mon --hosts $(ceph orch ps | grep mgr.y | awk '"'"'{print $2}'"'"')' 2024-03-26T02:47:21.848 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:21 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3596418030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:47:21.849 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:21 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3596418030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:47:22.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:21 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/3596418030' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:47:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:47:21] "GET /metrics HTTP/1.1" 200 38012 "" "Prometheus/2.43.0" 2024-03-26T02:47:22.721 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:22 smithi164 ceph-mon[24851]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:22.722 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:22 smithi164 ceph-mon[29512]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:22.958 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:22 smithi172 ceph-mon[67937]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:24.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:24 smithi172 ceph-mon[67937]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:24 smithi172 ceph-mon[67937]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:24.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:24 smithi172 ceph-mon[67937]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mon", "hosts": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:24.914 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:24 smithi164 ceph-mon[24851]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:24.915 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:24 smithi164 ceph-mon[24851]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:24.915 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:24 smithi164 ceph-mon[24851]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mon", "hosts": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:24.915 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:24 smithi164 ceph-mon[29512]: from='client.15684 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:24.915 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:24 smithi164 ceph-mon[29512]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:24.915 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:24 smithi164 ceph-mon[29512]: from='client.15690 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "mon", "hosts": "smithi164", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:25.882 INFO:teuthology.orchestra.run.smithi164.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:47:26.247 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:25 smithi164 ceph-mon[24851]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:26.247 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:25 smithi164 ceph-mon[29512]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:26.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:25 smithi172 ceph-mon[67937]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:26.571 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-26T02:47:27.162 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[24851]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:47:27.162 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:27.162 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[24851]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[29512]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:27.163 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:26 smithi164 ceph-mon[29512]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:47:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:26 smithi172 ceph-mon[67937]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:47:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:26 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:26 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:26 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:26 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:47:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:26 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:26 smithi172 ceph-mon[67937]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:47:28.218 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:27 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:28.218 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:27 smithi164 ceph-mon[29512]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:28.218 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:27 smithi164 ceph-mon[24851]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:28.218 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:27 smithi164 ceph-mon[24851]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:28.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:27 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:28.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:27 smithi172 ceph-mon[67937]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:29.970 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:47:30.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:30.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:47:30.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:47:30.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: Upgrade: Setting container_image for all mgr 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: Upgrade: It appears safe to stop mon.a 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[29512]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:47:30.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:47:30.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:30.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:30.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: Upgrade: Setting container_image for all mgr 2024-03-26T02:47:30.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:30.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:47:30.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-26T02:47:30.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: Upgrade: It appears safe to stop mon.a 2024-03-26T02:47:30.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:30 smithi164 ceph-mon[24851]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:30.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:30.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:47:30.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:47:30.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:30.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:30.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mgr 2024-03-26T02:47:30.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:30.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:47:30.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-03-26T02:47:30.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: Upgrade: It appears safe to stop mon.a 2024-03-26T02:47:30.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:30 smithi172 ceph-mon[67937]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (2m) 2m ago 24m 14.6M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (116s) 41s ago 23m 75.3M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (3m) 2m ago 23m 44.5M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (3m) 41s ago 28m 451M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (13m) 2m ago 30m 503M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (30m) 2m ago 30m 77.9M 2048M 17.2.0 e1d6a67b021e 664b72079ac2 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (46s) 41s ago 28m 21.6M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (29m) 2m ago 29m 55.5M 2048M 17.2.0 e1d6a67b021e e5d1a0686907 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (2m) 2m ago 24m 8682k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:47:30.652 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (2m) 41s ago 24m 16.3M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (28m) 2m ago 28m 54.9M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (27m) 2m ago 27m 56.5M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (27m) 2m ago 27m 53.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (27m) 2m ago 27m 57.2M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (26m) 41s ago 26m 56.2M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (26m) 41s ago 26m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (25m) 41s ago 25m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (25m) 41s ago 25m 55.5M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (2m) 41s ago 24m 65.6M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (23m) 2m ago 23m 77.0M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:47:30.653 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (23m) 41s ago 23m 79.8M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:47:31.058 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:47:31.058 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:47:31.058 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-26T02:47:31.058 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 1 2024-03-26T02:47:31.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:47:31.058 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:47:31.061 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:47:31.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:47:31.061 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:47:31.061 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:47:31.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:47:31.061 INFO:teuthology.orchestra.run.smithi164.stdout: "mds": {}, 2024-03-26T02:47:31.061 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:47:31.062 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:47:31.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:47:31.062 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:47:31.062 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-03-26T02:47:31.062 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:47:31.062 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:47:31.062 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:47:31.426 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:47:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:47:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:47:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi164", 2024-03-26T02:47:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:47:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "0/2 daemons upgraded", 2024-03-26T02:47:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading mon daemons", 2024-03-26T02:47:31.427 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:47:31.427 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:47:31.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[29512]: from='client.25600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[29512]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.525 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[29512]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[29512]: from='client.? 172.21.15.164:0/3227312771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[24851]: from='client.25600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[24851]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[24851]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:47:31.526 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[24851]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:31.527 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:31 smithi164 ceph-mon[24851]: from='client.? 172.21.15.164:0/3227312771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:31 smithi172 ceph-mon[67937]: from='client.25600 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:31 smithi172 ceph-mon[67937]: from='client.15702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:31 smithi172 ceph-mon[67937]: from='client.25612 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:31 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' 2024-03-26T02:47:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:31 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:47:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:31 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:47:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:31 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:31 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/3227312771' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:32.194 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:47:31] "GET /metrics HTTP/1.1" 200 38012 "" "Prometheus/2.43.0" 2024-03-26T02:47:32.528 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-mon[24851]: Upgrade: Updating mon.a 2024-03-26T02:47:32.528 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-mon[24851]: Deploying daemon mon.a on smithi164 2024-03-26T02:47:32.528 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-mon[24851]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:32.528 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-mon[24851]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:32.529 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-mon[29512]: Upgrade: Updating mon.a 2024-03-26T02:47:32.529 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-mon[29512]: Deploying daemon mon.a on smithi164 2024-03-26T02:47:32.529 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-mon[29512]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:32.529 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:32 smithi164 ceph-mon[29512]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:32.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:32 smithi172 ceph-mon[67937]: Upgrade: Updating mon.a 2024-03-26T02:47:32.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:32 smithi172 ceph-mon[67937]: Deploying daemon mon.a on smithi164 2024-03-26T02:47:32.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:32 smithi172 ceph-mon[67937]: from='client.15720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:47:32.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:32 smithi172 ceph-mon[67937]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:47:33.083 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:32 smithi164 systemd[1]: Stopping Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a[24847]: 2024-03-26T02:47:33.082+0000 7f086433b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a[24847]: 2024-03-26T02:47:33.082+0000 7f086433b700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:33 smithi164 podman[83451]: 2024-03-26 02:47:33.204665358 +0000 UTC m=+0.238363563 container died 664b72079ac2f4a96b2b33346b246e146fa3934fe6fa63dfc8df6c364bb84a4c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.buildah.version=1.19.8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, distribution-scope=public, name=centos-stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-type=git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:33] ENGINE Bus STOPPING 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:33] ENGINE Bus STOPPED 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:33] ENGINE Bus STARTING 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:33] ENGINE Serving on http://:::9283 2024-03-26T02:47:33.460 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:33] ENGINE Bus STARTED 2024-03-26T02:47:33.759 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:33 smithi164 podman[83451]: 2024-03-26 02:47:33.540963202 +0000 UTC m=+0.574661402 container cleanup 664b72079ac2f4a96b2b33346b246e146fa3934fe6fa63dfc8df6c364bb84a4c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, name=centos-stream, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., distribution-scope=public, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True) 2024-03-26T02:47:33.759 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:33 smithi164 bash[83451]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a 2024-03-26T02:47:34.018 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:33 smithi164 podman[83476]: 2024-03-26 02:47:33.75890742 +0000 UTC m=+0.546250704 container remove 664b72079ac2f4a96b2b33346b246e146fa3934fe6fa63dfc8df6c364bb84a4c (image=quay.io/ceph/ceph:v17.2.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, com.redhat.component=centos-stream-container, version=8) 2024-03-26T02:47:34.018 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T02:47:34.018 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 systemd[1]: Stopped Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:47:34.018 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service: Consumed 25.858s CPU time. 2024-03-26T02:47:34.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 systemd[1]: Starting Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:47:34.680 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 podman[83603]: 2024-03-26 02:47:34.401794131 +0000 UTC m=+0.100926976 container create 94b7005f8327c28c8df1c00c912fbfb64b7c89218646bf74a7e9c162a3eb9c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 podman[83603]: 2024-03-26 02:47:34.319137713 +0000 UTC m=+0.018270514 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 podman[83603]: 2024-03-26 02:47:34.604165142 +0000 UTC m=+0.303297942 container init 94b7005f8327c28c8df1c00c912fbfb64b7c89218646bf74a7e9c162a3eb9c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 podman[83603]: 2024-03-26 02:47:34.607321183 +0000 UTC m=+0.306453984 container start 94b7005f8327c28c8df1c00c912fbfb64b7c89218646bf74a7e9c162a3eb9c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev), process ceph-mon, pid 2 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: pidfile_write: ignore empty --pid-file 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: load: jerasure load: lrc 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: RocksDB version: 7.9.2 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Git sha 0 2024-03-26T02:47:34.681 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Compile date 2024-03-25 23:25:11 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: DB SUMMARY 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: DB Session ID: 0MEI8XRTHQRB47TGELBT 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: MANIFEST file: MANIFEST-000015 size: 3094 Bytes 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000063.sst 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000061.log size: 121722 ; 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T02:47:34.682 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.env: 0x559d998d3c60 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.info_log: 0x559d9af3fae0 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.statistics: (nil) 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.use_fsync: 0 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:47:34.683 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.db_log_dir: 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.wal_dir: 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.write_buffer_manager: 0x559d9af10820 2024-03-26T02:47:34.684 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.unordered_write: 0 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.row_cache: None 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.wal_filter: None 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:47:34.685 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.wal_compression: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:47:34.686 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_open_files: -1 2024-03-26T02:47:34.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Compression algorithms supported: 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: kZSTD supported: 0 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:47:34.689 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:47:34.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:47:34.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T02:47:34.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-03-26T02:47:34.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T02:47:34.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:47:34.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.merge_operator: 2024-03-26T02:47:34.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_filter: None 2024-03-26T02:47:34.690 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559d9af3fcc0) 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: index_type: 0 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: data_block_index_type: 0 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: index_shortening: 1 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: checksum: 4 2024-03-26T02:47:34.691 INFO:journalctl@ceph.mon.a.smithi164.stdout: no_block_cache: 0 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_cache: 0x559d9af43090 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_cache_options: 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: capacity : 536870912 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: num_shard_bits : 4 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: strict_capacity_limit : 0 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_cache_compressed: (nil) 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: persistent_cache: (nil) 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_size: 4096 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_size_deviation: 10 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_restart_interval: 16 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: index_block_restart_interval: 1 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: metadata_block_size: 4096 2024-03-26T02:47:34.692 INFO:journalctl@ceph.mon.a.smithi164.stdout: partition_filters: 0 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: use_delta_encoding: 1 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: filter_policy: bloomfilter 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: whole_key_filtering: 1 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: verify_compression: 0 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: format_version: 5 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: enable_index_compression: 1 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: block_align: 0 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: max_auto_readahead_size: 262144 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: prepopulate_block_cache: 0 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: initial_auto_readahead_size: 8192 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:47:34.693 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression: NoCompression 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.num_levels: 7 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:47:34.694 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:47:34.695 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T02:47:34.696 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T02:47:34.697 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.ttl: 2592000 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T02:47:34.698 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 63.sst 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 65, last_sequence is 29229, log_number is 61,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 61 2024-03-26T02:47:34.699 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 400de2b8-7a86-4219-b836-e9b955f4ef23 2024-03-26T02:47:34.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421254637116, "job": 1, "event": "recovery_started", "wal_files": [61]} 2024-03-26T02:47:34.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #61 mode 2 2024-03-26T02:47:34.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421254637976, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 66, "file_size": 120202, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 29231, "largest_seqno": 29378, "table_properties": {"data_size": 118543, "index_size": 365, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 261, "raw_key_size": 2211, "raw_average_key_size": 25, "raw_value_size": 116688, "raw_average_value_size": 1356, "num_data_blocks": 16, "num_entries": 86, "num_filter_entries": 86, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711421254, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "400de2b8-7a86-4219-b836-e9b955f4ef23", "db_session_id": "0MEI8XRTHQRB47TGELBT", "orig_file_number": 66, "seqno_to_time_mapping": "N/A"}} 2024-03-26T02:47:34.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421254638057, "job": 1, "event": "recovery_finished"} 2024-03-26T02:47:34.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/version_set.cc:5047] Creating manifest 68 2024-03-26T02:47:34.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T02:47:34.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000061.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:47:34.700 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559d9b032000 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: DB pointer 0x559d9b01c000 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** DB Stats ** 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-26T02:47:34.701 INFO:journalctl@ceph.mon.a.smithi164.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: L0 1/0 117.38 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 212.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: L6 1/0 9.40 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: Sum 2/0 9.51 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 212.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 212.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 212.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:47:34.702 INFO:journalctl@ceph.mon.a.smithi164.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: Cumulative compaction: 0.00 GB write, 34.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: Interval compaction: 0.00 GB write, 34.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: Block cache BinnedLRUCache@0x559d9af43090#2 capacity: 512.00 MB usage: 0.77 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.31 KB,5.96046e-05%) IndexBlock(1,0.45 KB,8.64267e-05%) Misc(1,0.00 KB,0%) 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: starting mon.a rank 0 at public addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] at bind addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: mon.a@-1(???) e3 preinit fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: mon.a@-1(???).mds e1 new map 2024-03-26T02:47:34.703 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: mon.a@-1(???).mds e1 print_map 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout: e1 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout: legacy client fscid: -1 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout: 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout: No filesystems configured 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: mon.a@-1(???).osd e111 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: mon.a@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: mon.a@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: mon.a@-1(???).osd e111 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 ceph-mon[83622]: mon.a@-1(???).paxosservice(auth 1..27) refresh upgraded, format 0 -> 3 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 bash[83603]: 94b7005f8327c28c8df1c00c912fbfb64b7c89218646bf74a7e9c162a3eb9c7b 2024-03-26T02:47:34.704 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:34 smithi164 systemd[1]: Started Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:47:38.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ignoring --setuser ceph since I am not root 2024-03-26T02:47:38.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ignoring --setgroup ceph since I am not root 2024-03-26T02:47:38.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mgr[62330]: -- 172.21.15.164:0/542506579 <== mon.0 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564e8b82e5a0 con 0x564e8b803000 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: mon.a calling monitor election 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: monmap epoch 3 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:47:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: last_changed 2024-03-26T02:18:39.946932+0000 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: created 2024-03-26T02:17:11.294549+0000 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: min_mon_release 17 (quincy) 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: election_strategy: 1 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: 0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: 1: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: 2: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: fsmap 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: mgrmap e40: y(active, since 3m), standbys: x 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: overall HEALTH_OK 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: from='mgr.25192 ' entity='' 2024-03-26T02:47:38.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[83622]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: mon.a calling monitor election 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: monmap epoch 3 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: last_changed 2024-03-26T02:18:39.946932+0000 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: created 2024-03-26T02:17:11.294549+0000 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: min_mon_release 17 (quincy) 2024-03-26T02:47:38.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: election_strategy: 1 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: 0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: 1: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: 2: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: fsmap 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: mgrmap e40: y(active, since 3m), standbys: x 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: overall HEALTH_OK 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: from='mgr.25192 ' entity='' 2024-03-26T02:47:38.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:37 smithi164 ceph-mon[29512]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: ignoring --setuser ceph since I am not root 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: ignoring --setgroup ceph since I am not root 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mgr[61353]: -- 172.21.15.172:0/1573006324 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55be53fc85a0 con 0x55be53f9d000 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: from='mgr.25192 172.21.15.164:0/2720349553' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: mon.a calling monitor election 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: monmap epoch 3 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: last_changed 2024-03-26T02:18:39.946932+0000 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: created 2024-03-26T02:17:11.294549+0000 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: min_mon_release 17 (quincy) 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: election_strategy: 1 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: 0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: 1: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: 2: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: fsmap 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:47:38.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: mgrmap e40: y(active, since 3m), standbys: x 2024-03-26T02:47:38.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: overall HEALTH_OK 2024-03-26T02:47:38.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: from='mgr.25192 ' entity='' 2024-03-26T02:47:38.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:37 smithi172 ceph-mon[67937]: mgrmap e41: y(active, since 3m), standbys: x 2024-03-26T02:47:38.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:38 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:38.259+0000 7f6e28f00200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:47:38.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:38 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:38.351+0000 7f6e28f00200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:47:38.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:38.261+0000 7f58dc054200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:47:38.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:38.356+0000 7f58dc054200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:47:39.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:38 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:38.988+0000 7f6e28f00200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:47:39.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:39.080+0000 7f6e28f00200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:47:39.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:39.221+0000 7f6e28f00200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:47:39.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:38.995+0000 7f58dc054200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:47:39.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:39.089+0000 7f58dc054200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:47:39.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:39.231+0000 7f58dc054200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:47:39.666 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:39.316+0000 7f6e28f00200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:47:39.667 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:39.398+0000 7f6e28f00200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:47:39.683 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:39.327+0000 7f58dc054200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:47:39.683 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:39.411+0000 7f58dc054200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:47:39.683 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:39.681+0000 7f58dc054200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:47:39.956 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-mon[29512]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-26T02:47:39.956 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-mon[83622]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-26T02:47:39.957 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:39.666+0000 7f6e28f00200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:47:39.957 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:39.777+0000 7f6e28f00200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:47:39.957 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:39.863+0000 7f6e28f00200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:47:39.957 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:39 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:39.955+0000 7f6e28f00200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:47:39.976 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-mon[67937]: mgrmap e42: y(active, since 3m), standbys: x 2024-03-26T02:47:39.976 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:39.793+0000 7f58dc054200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:47:39.976 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:39.881+0000 7f58dc054200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:47:39.976 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:39.975+0000 7f58dc054200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:47:40.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:40.132+0000 7f6e28f00200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:47:40.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:40.152+0000 7f58dc054200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:47:41.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:41.094+0000 7f6e28f00200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:47:41.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:41.227+0000 7f6e28f00200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:47:41.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:41.117+0000 7f58dc054200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:47:41.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:41.248+0000 7f58dc054200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:47:41.859 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:41.569+0000 7f6e28f00200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:47:41.859 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:41.659+0000 7f6e28f00200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:47:41.859 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:41.858+0000 7f6e28f00200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:47:41.879 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:41.587+0000 7f58dc054200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:47:41.879 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:41.677+0000 7f58dc054200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:47:42.217 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:42.044+0000 7f6e28f00200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:47:42.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:41.879+0000 7f58dc054200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:47:42.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:42.071+0000 7f58dc054200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:47:42.544 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:42.341+0000 7f6e28f00200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:47:42.626 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:42.363+0000 7f58dc054200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:47:42.872 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:42.619+0000 7f6e28f00200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:47:42.872 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:42.871+0000 7f6e28f00200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:47:43.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:42.625+0000 7f58dc054200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:47:43.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:42.868+0000 7f58dc054200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:47:43.707 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:47:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:47:43.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi164.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:47:43.707 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:47:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:47:43.518Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-26T02:47:44.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:43.706+0000 7f6e28f00200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:47:44.056 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:43.685+0000 7f58dc054200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:47:44.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:44.282+0000 7f6e28f00200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:47:44.546 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:44.383+0000 7f6e28f00200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:47:44.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:44.262+0000 7f58dc054200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:47:44.556 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:44.363+0000 7f58dc054200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:47:44.845 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:44.588+0000 7f58dc054200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:47:44.845 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: [26/Mar/2024:02:47:44] ENGINE Bus STARTING 2024-03-26T02:47:44.845 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: CherryPy Checker: 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: The Application mounted at '' has an empty config. 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: [26/Mar/2024:02:47:44] ENGINE Serving on http://:::9283 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x[61349]: [26/Mar/2024:02:47:44] ENGINE Bus STARTED 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: Standby manager daemon x restarted 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: Standby manager daemon x started 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:47:44.846 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:47:44.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: Active manager daemon y restarted 2024-03-26T02:47:44.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: Activating manager daemon y 2024-03-26T02:47:44.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:47:44.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: mgrmap e43: y(active, starting, since 0.00738661s), standbys: x 2024-03-26T02:47:44.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:47:44.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:47:44.847 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:47:44.848 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:47:44.859 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:44.605+0000 7f6e28f00200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:47:44.860 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:44] ENGINE Bus STARTING 2024-03-26T02:47:44.860 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: Standby manager daemon x restarted 2024-03-26T02:47:44.860 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: Standby manager daemon x started 2024-03-26T02:47:44.861 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:47:44.861 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:47:44.861 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:47:44.861 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:47:44.861 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: Active manager daemon y restarted 2024-03-26T02:47:44.862 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: Activating manager daemon y 2024-03-26T02:47:44.862 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:47:44.862 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: mgrmap e43: y(active, starting, since 0.00738661s), standbys: x 2024-03-26T02:47:44.862 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:47:44.862 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:47:44.862 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:47:44.863 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:47:44.863 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:47:44.863 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:47:44.863 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:47:44.863 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:47:44.863 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:47:44.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:47:44.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:47:44.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:47:44.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:47:44.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:47:44.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:47:44.865 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:47:44.865 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:47:44.763Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify success" attempts=3 2024-03-26T02:47:44.865 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: Standby manager daemon x restarted 2024-03-26T02:47:44.866 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: Standby manager daemon x started 2024-03-26T02:47:44.866 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:47:44.866 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:47:44.866 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:47:44.866 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.? 172.21.15.172:0/3709161458' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:47:44.866 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: Active manager daemon y restarted 2024-03-26T02:47:44.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: Activating manager daemon y 2024-03-26T02:47:44.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:47:44.867 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: mgrmap e43: y(active, starting, since 0.00738661s), standbys: x 2024-03-26T02:47:44.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:47:44.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:47:44.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:47:44.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:47:44.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:47:44.868 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:47:44.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:47:44.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:47:44.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:47:44.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:47:44.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:47:44.869 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:47:44.870 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:47:44.870 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:47:44.870 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:47:44.870 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:47:45.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: CherryPy Checker: 2024-03-26T02:47:45.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: The Application mounted at '' has an empty config. 2024-03-26T02:47:45.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:45.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:44] ENGINE Serving on http://:::9283 2024-03-26T02:47:45.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:44] ENGINE Bus STARTED 2024-03-26T02:47:45.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:45 smithi172 ceph-mon[67937]: Manager daemon y is now available 2024-03-26T02:47:45.924 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:45.924 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:45.924 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:47:45.924 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:45 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:47:45.924 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:47:45.924 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:45 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:47:45.998 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:47:45.666+0000 7f6ddf0a2700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-03-26T02:47:45.999 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[29512]: Manager daemon y is now available 2024-03-26T02:47:45.999 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:45.999 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:45.999 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:47:45.999 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:47:45.999 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:47:45.999 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:47:46.000 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:47:45.628Z caller=notify.go:743 level=info component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify success" attempts=4 2024-03-26T02:47:46.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[83622]: Manager daemon y is now available 2024-03-26T02:47:46.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:46.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:47:46.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:47:46.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:47:46.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:47:46.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:45 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:47:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:46 smithi172 ceph-mon[67937]: [26/Mar/2024:02:47:45] ENGINE Bus STARTING 2024-03-26T02:47:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:46 smithi172 ceph-mon[67937]: [26/Mar/2024:02:47:45] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:47:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:46 smithi172 ceph-mon[67937]: [26/Mar/2024:02:47:45] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:47:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:46 smithi172 ceph-mon[67937]: [26/Mar/2024:02:47:45] ENGINE Bus STARTED 2024-03-26T02:47:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:46 smithi172 ceph-mon[67937]: mgrmap e44: y(active, since 1.04742s), standbys: x 2024-03-26T02:47:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:46 smithi172 ceph-mon[67937]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:46.903 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[83622]: [26/Mar/2024:02:47:45] ENGINE Bus STARTING 2024-03-26T02:47:46.903 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[83622]: [26/Mar/2024:02:47:45] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:47:46.903 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[83622]: [26/Mar/2024:02:47:45] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:47:46.903 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[83622]: [26/Mar/2024:02:47:45] ENGINE Bus STARTED 2024-03-26T02:47:46.903 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[83622]: mgrmap e44: y(active, since 1.04742s), standbys: x 2024-03-26T02:47:46.903 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[83622]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:46.904 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[29512]: [26/Mar/2024:02:47:45] ENGINE Bus STARTING 2024-03-26T02:47:46.904 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[29512]: [26/Mar/2024:02:47:45] ENGINE Serving on https://172.21.15.164:7150 2024-03-26T02:47:46.904 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[29512]: [26/Mar/2024:02:47:45] ENGINE Serving on http://172.21.15.164:8765 2024-03-26T02:47:46.904 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[29512]: [26/Mar/2024:02:47:45] ENGINE Bus STARTED 2024-03-26T02:47:46.904 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[29512]: mgrmap e44: y(active, since 1.04742s), standbys: x 2024-03-26T02:47:46.904 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:46 smithi164 ceph-mon[29512]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:47.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:47 smithi164 ceph-mon[29512]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:47.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:47 smithi164 ceph-mon[29512]: mgrmap e45: y(active, since 2s), standbys: x 2024-03-26T02:47:47.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:47 smithi164 ceph-mon[83622]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:47.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:47 smithi164 ceph-mon[83622]: mgrmap e45: y(active, since 2s), standbys: x 2024-03-26T02:47:47.978 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:47 smithi172 ceph-mon[67937]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:47.978 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:47 smithi172 ceph-mon[67937]: mgrmap e45: y(active, since 2s), standbys: x 2024-03-26T02:47:48.962 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:48 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:48.963 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:48 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:48 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:50.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[29512]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[29512]: mgrmap e46: y(active, since 4s), standbys: x 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[83622]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[83622]: mgrmap e46: y(active, since 4s), standbys: x 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:49 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:49 smithi172 ceph-mon[67937]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:49 smithi172 ceph-mon[67937]: mgrmap e46: y(active, since 4s), standbys: x 2024-03-26T02:47:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:49 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:49 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:49 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:49 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[83622]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:51.419 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.419 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:51 smithi164 ceph-mon[29512]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:51.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:51 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:51 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:51 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:51 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:51.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:51 smithi172 ceph-mon[67937]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:47:52.256 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:47:52] "GET /metrics HTTP/1.1" 200 35214 "" "Prometheus/2.43.0" 2024-03-26T02:47:53.750 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:53 smithi172 ceph-mon[67937]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:47:53.750 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:53 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:53.750 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:53 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:53.930 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:53 smithi164 ceph-mon[29512]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:47:53.931 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:53 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:53.931 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:53 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:53.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:53 smithi164 ceph-mon[83622]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:47:53.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:53 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:53.931 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:53 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Detected new or changed devices on smithi164 2024-03-26T02:47:55.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Detected new or changed devices on smithi164 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:47:55.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.297 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:55 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Detected new or changed devices on smithi164 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Updating smithi164:/etc/ceph/ceph.conf 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Updating smithi172:/etc/ceph/ceph.conf 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Updating smithi164:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:47:55.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Updating smithi172:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:47:55.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:47:55.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Updating smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:47:55.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: Updating smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:47:55.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:55.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:55 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[83622]: Upgrade: It appears safe to stop mon.c 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-26T02:47:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:56 smithi164 ceph-mon[29512]: Upgrade: It appears safe to stop mon.c 2024-03-26T02:47:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:56 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:47:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:56 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:47:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:56 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-03-26T02:47:56.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:56 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-03-26T02:47:56.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:56 smithi172 ceph-mon[67937]: Upgrade: It appears safe to stop mon.c 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[83622]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[83622]: Upgrade: Updating mon.c 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[83622]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[83622]: Deploying daemon mon.c on smithi164 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[29512]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:47:57.947 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[29512]: Upgrade: Updating mon.c 2024-03-26T02:47:57.948 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[29512]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:57.948 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:47:57.948 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:47:57.948 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[29512]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:57.948 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:57 smithi164 ceph-mon[29512]: Deploying daemon mon.c on smithi164 2024-03-26T02:47:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:57 smithi172 ceph-mon[67937]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:47:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:57 smithi172 ceph-mon[67937]: Upgrade: Updating mon.c 2024-03-26T02:47:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:57 smithi172 ceph-mon[67937]: from='mgr.25636 ' entity='mgr.y' 2024-03-26T02:47:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:47:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:47:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:47:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:57 smithi172 ceph-mon[67937]: Deploying daemon mon.c on smithi164 2024-03-26T02:47:58.786 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:58.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:58 smithi164 systemd[1]: Stopping Ceph mon.c for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:47:58.786 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-mon[29512]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:59.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c[29508]: 2024-03-26T02:47:58.783+0000 7f0893171700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:47:59.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c[29508]: 2024-03-26T02:47:58.784+0000 7f0893171700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T02:47:59.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:58 smithi164 podman[88440]: 2024-03-26 02:47:58.80563639 +0000 UTC m=+0.132010635 container died e5d1a0686907b2fdb2650c27e8722f45dc7af372a994a459332b7914e73ee114 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, GIT_CLEAN=True, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-03-26T02:47:59.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:58] ENGINE Bus STOPPING 2024-03-26T02:47:59.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:58] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:47:59.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:58] ENGINE Bus STOPPED 2024-03-26T02:47:59.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:58] ENGINE Bus STARTING 2024-03-26T02:47:59.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:58] ENGINE Serving on http://:::9283 2024-03-26T02:47:59.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:47:58 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:47:58] ENGINE Bus STARTED 2024-03-26T02:47:59.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:47:58 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:47:59.462 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:59 smithi164 podman[88440]: 2024-03-26 02:47:59.193173645 +0000 UTC m=+0.519547881 container cleanup e5d1a0686907b2fdb2650c27e8722f45dc7af372a994a459332b7914e73ee114 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , name=centos-stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, ceph=True) 2024-03-26T02:47:59.462 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:59 smithi164 bash[88440]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c 2024-03-26T02:47:59.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:59 smithi164 podman[88455]: 2024-03-26 02:47:59.461450715 +0000 UTC m=+0.650814863 container remove e5d1a0686907b2fdb2650c27e8722f45dc7af372a994a459332b7914e73ee114 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, version=8, architecture=x86_64, com.redhat.component=centos-stream-container, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.buildah.version=1.19.8, ceph=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, vendor=Red Hat, Inc.) 2024-03-26T02:47:59.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:59 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.c.service: Deactivated successfully. 2024-03-26T02:47:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:59 smithi164 systemd[1]: Stopped Ceph mon.c for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:47:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:59 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.c.service: Consumed 14.931s CPU time. 2024-03-26T02:48:00.192 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:47:59 smithi164 systemd[1]: Starting Ceph mon.c for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:48:00.192 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 podman[88600]: 2024-03-26 02:47:59.938200554 +0000 UTC m=+0.018147039 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:48:00.192 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 podman[88600]: 2024-03-26 02:48:00.04595645 +0000 UTC m=+0.125902984 container create 4909686c6b24901873cb1c278d6da7dbfce2c218eb2508470a84fb599453677c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T02:48:00.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 podman[88600]: 2024-03-26 02:48:00.298492955 +0000 UTC m=+0.378439433 container init 4909686c6b24901873cb1c278d6da7dbfce2c218eb2508470a84fb599453677c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 podman[88600]: 2024-03-26 02:48:00.301708291 +0000 UTC m=+0.381654775 container start 4909686c6b24901873cb1c278d6da7dbfce2c218eb2508470a84fb599453677c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev), process ceph-mon, pid 2 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: pidfile_write: ignore empty --pid-file 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: load: jerasure load: lrc 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: RocksDB version: 7.9.2 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Git sha 0 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Compile date 2024-03-25 23:25:11 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: DB SUMMARY 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: DB Session ID: 0OJF3LWLPBBSFI8J4JSL 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: CURRENT file: CURRENT 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: MANIFEST file: MANIFEST-000009 size: 3209 Bytes 2024-03-26T02:48:00.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000060.sst 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000058.log size: 2444270 ; 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.error_if_exists: 0 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.create_if_missing: 0 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.paranoid_checks: 1 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.env: 0x555855a25c60 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.info_log: 0x555856c9fae0 2024-03-26T02:48:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.statistics: (nil) 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.use_fsync: 0 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_log_file_size: 0 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.allow_fallocate: 1 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.use_direct_reads: 0 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T02:48:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.db_log_dir: 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.wal_dir: 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.write_buffer_manager: 0x555856c70820 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T02:48:00.549 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.unordered_write: 0 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.row_cache: None 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.wal_filter: None 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T02:48:00.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.two_write_queues: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.wal_compression: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.atomic_flush: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.log_readahead_size: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T02:48:00.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_background_jobs: 2 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_background_compactions: -1 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_subcompactions: 1 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_open_files: -1 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T02:48:00.552 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_background_flushes: -1 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Compression algorithms supported: 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: kZSTD supported: 0 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: kXpressCompression supported: 0 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: kZlibCompression supported: 1 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: kSnappyCompression supported: 1 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: kLZ4Compression supported: 1 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: kBZip2Compression supported: 0 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T02:48:00.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.merge_operator: 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_filter: None 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555856c9fcc0) 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout: cache_index_and_filter_blocks: 1 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T02:48:00.554 INFO:journalctl@ceph.mon.c.smithi164.stdout: pin_top_level_index_and_filter: 1 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_type: 0 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: data_block_index_type: 0 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_shortening: 1 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: checksum: 4 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: no_block_cache: 0 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache: 0x555856ca3090 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_name: BinnedLRUCache 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_options: 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: capacity : 536870912 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: num_shard_bits : 4 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: strict_capacity_limit : 0 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: high_pri_pool_ratio: 0.000 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_cache_compressed: (nil) 2024-03-26T02:48:00.555 INFO:journalctl@ceph.mon.c.smithi164.stdout: persistent_cache: (nil) 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_size: 4096 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_size_deviation: 10 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_restart_interval: 16 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: index_block_restart_interval: 1 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: metadata_block_size: 4096 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: partition_filters: 0 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: use_delta_encoding: 1 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: filter_policy: bloomfilter 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: whole_key_filtering: 1 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: verify_compression: 0 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: read_amp_bytes_per_bit: 0 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: format_version: 5 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: enable_index_compression: 1 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: block_align: 0 2024-03-26T02:48:00.556 INFO:journalctl@ceph.mon.c.smithi164.stdout: max_auto_readahead_size: 262144 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout: prepopulate_block_cache: 0 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout: initial_auto_readahead_size: 8192 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression: NoCompression 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.num_levels: 7 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T02:48:00.557 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T02:48:00.558 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T02:48:00.559 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T02:48:00.560 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.inplace_update_support: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.bloom_locality: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.max_successive_merges: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T02:48:00.561 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.ttl: 2592000 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.enable_blob_files: false 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.min_blob_size: 0 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 60.sst 2024-03-26T02:48:00.562 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 62, last_sequence is 32198, log_number is 58,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 58 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4774027b-b1e2-45e2-8b9a-904afc2dbfd9 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421280331835, "job": 1, "event": "recovery_started", "wal_files": [58]} 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #58 mode 2 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421280337867, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 63, "file_size": 1542792, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 32203, "largest_seqno": 32620, "table_properties": {"data_size": 1539865, "index_size": 1375, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4812, "raw_average_key_size": 24, "raw_value_size": 1535362, "raw_average_value_size": 7676, "num_data_blocks": 62, "num_entries": 200, "num_filter_entries": 200, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711421280, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4774027b-b1e2-45e2-8b9a-904afc2dbfd9", "db_session_id": "0OJF3LWLPBBSFI8J4JSL", "orig_file_number": 63, "seqno_to_time_mapping": "N/A"}} 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711421280337962, "job": 1, "event": "recovery_finished"} 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/version_set.cc:5047] Creating manifest 65 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000058.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x555856d92000 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: DB pointer 0x555856d7c000 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T02:48:00.563 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** DB Stats ** 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: L0 1/0 1.47 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 525.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: L6 1/0 9.92 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Sum 2/0 11.39 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 525.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-26T02:48:00.564 INFO:journalctl@ceph.mon.c.smithi164.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 525.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** Compaction Stats [default] ** 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 525.1 0.00 0.00 1 0.003 0 0 0.0 0.0 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: Cumulative compaction: 0.00 GB write, 175.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:48:00.565 INFO:journalctl@ceph.mon.c.smithi164.stdout: Interval compaction: 0.00 GB write, 175.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: Block cache BinnedLRUCache@0x555856ca3090#2 capacity: 512.00 MB usage: 2.05 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.45 KB,0.000277162%) Misc(1,0.00 KB,0%) 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: starting mon.c rank 1 at public addrs [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] at bind addrs [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: mon.c@-1(???) e3 preinit fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: mon.c@-1(???).mds e1 new map 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: mon.c@-1(???).mds e1 print_map 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: e1 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: legacy client fscid: -1 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: 2024-03-26T02:48:00.566 INFO:journalctl@ceph.mon.c.smithi164.stdout: No filesystems configured 2024-03-26T02:48:00.567 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: mon.c@-1(???).osd e112 crush map has features 3314933000854323200, adjusting msgr requires 2024-03-26T02:48:00.567 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: mon.c@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:48:00.567 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: mon.c@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:48:00.567 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: mon.c@-1(???).osd e112 crush map has features 432629239337189376, adjusting msgr requires 2024-03-26T02:48:00.567 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 ceph-mon[88618]: mon.c@-1(???).paxosservice(auth 1..28) refresh upgraded, format 0 -> 3 2024-03-26T02:48:00.567 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 bash[88600]: 4909686c6b24901873cb1c278d6da7dbfce2c218eb2508470a84fb599453677c 2024-03-26T02:48:00.567 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:00 smithi164 systemd[1]: Started Ceph mon.c for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:48:01.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:01 smithi164 ceph-mon[88618]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:48:01.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:48:01.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:48:01.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:48:01.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:01 smithi164 ceph-mon[83622]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:48:01.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:48:01.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:48:01.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:48:01.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:01 smithi172 ceph-mon[67937]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:48:01.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:48:01.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:48:01.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:48:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:48:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:48:02] "GET /metrics HTTP/1.1" 200 38015 "" "Prometheus/2.43.0" 2024-03-26T02:48:05.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:48:05 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:48:05.347+0000 7f6ddf0a2700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-26T02:48:06.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: overall HEALTH_OK 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: mon.a calling monitor election 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: mon.c calling monitor election 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: monmap epoch 4 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: last_changed 2024-03-26T02:48:01.089832+0000 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: created 2024-03-26T02:17:11.294549+0000 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: min_mon_release 19 (squid) 2024-03-26T02:48:06.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: election_strategy: 1 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: 0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: 1: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: 2: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: fsmap 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: mgrmap e46: y(active, since 21s), standbys: x 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: overall HEALTH_OK 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: overall HEALTH_OK 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: mon.a calling monitor election 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: mon.c calling monitor election 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: monmap epoch 4 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:48:06.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: last_changed 2024-03-26T02:48:01.089832+0000 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: created 2024-03-26T02:17:11.294549+0000 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: min_mon_release 19 (squid) 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: election_strategy: 1 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: 0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: 1: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: 2: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: fsmap 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: mgrmap e46: y(active, since 21s), standbys: x 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: overall HEALTH_OK 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:06.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:07.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:07 smithi164 ceph-mon[83622]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:07.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:07 smithi164 ceph-mon[88618]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:07.798 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:48:08.184 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:07 smithi172 ceph-mon[67937]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:08.499 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: mon.b calling monitor election 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: mon.c calling monitor election 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: mon.a calling monitor election 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: monmap epoch 4 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: last_changed 2024-03-26T02:48:01.089832+0000 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: created 2024-03-26T02:17:11.294549+0000 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: min_mon_release 19 (squid) 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: election_strategy: 1 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: 0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: 1: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: 2: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-26T02:48:08.500 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: fsmap 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: mgrmap e46: y(active, since 23s), standbys: x 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[83622]: overall HEALTH_OK 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: mon.b calling monitor election 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: mon.c calling monitor election 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: mon.a calling monitor election 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: monmap epoch 4 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: last_changed 2024-03-26T02:48:01.089832+0000 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: created 2024-03-26T02:17:11.294549+0000 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: min_mon_release 19 (squid) 2024-03-26T02:48:08.501 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: election_strategy: 1 2024-03-26T02:48:08.502 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: 0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-03-26T02:48:08.502 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: 1: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-26T02:48:08.502 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: 2: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-26T02:48:08.502 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: fsmap 2024-03-26T02:48:08.502 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:48:08.502 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: mgrmap e46: y(active, since 23s), standbys: x 2024-03-26T02:48:08.502 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:08 smithi164 ceph-mon[88618]: overall HEALTH_OK 2024-03-26T02:48:08.506 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:48:08.506 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (2m) 19s ago 24m 17.5M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:48:08.506 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (2m) 19s ago 24m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:48:08.506 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (3m) 19s ago 23m 49.6M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (3m) 19s ago 29m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (13m) 19s ago 30m 475M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (34s) 19s ago 30m 29.1M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (84s) 19s ago 29m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 starting - - - 2048M 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (3m) 19s ago 25m 7830k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (3m) 19s ago 25m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (28m) 19s ago 28m 54.8M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (28m) 19s ago 28m 56.7M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (28m) 19s ago 28m 53.5M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (27m) 19s ago 27m 57.4M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (27m) 19s ago 27m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:48:08.507 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (26m) 19s ago 26m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:48:08.508 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (26m) 19s ago 26m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:48:08.508 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (26m) 19s ago 26m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:48:08.508 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (3m) 19s ago 25m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:48:08.508 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (23m) 19s ago 23m 77.8M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:48:08.508 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (23m) 19s ago 23m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:48:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: mon.b calling monitor election 2024-03-26T02:48:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: mon.c calling monitor election 2024-03-26T02:48:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: mon.a calling monitor election 2024-03-26T02:48:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-26T02:48:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: monmap epoch 4 2024-03-26T02:48:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: fsid e2654d3c-eb16-11ee-95d0-87774f69a715 2024-03-26T02:48:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: last_changed 2024-03-26T02:48:01.089832+0000 2024-03-26T02:48:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: created 2024-03-26T02:17:11.294549+0000 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: min_mon_release 19 (squid) 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: election_strategy: 1 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: 0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: 1: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: 2: [v2:172.21.15.172:3300/0,v1:172.21.15.172:6789/0] mon.b 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: fsmap 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: mgrmap e46: y(active, since 23s), standbys: x 2024-03-26T02:48:08.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:08 smithi172 ceph-mon[67937]: overall HEALTH_OK 2024-03-26T02:48:08.914 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:48:08.915 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-26T02:48:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 5 2024-03-26T02:48:08.916 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:48:08.916 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:48:09.275 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:48:09.275 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:48:09.275 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:48:09.275 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) mon on host(s) smithi164", 2024-03-26T02:48:09.275 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:48:09.275 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "1/2 daemons upgraded", 2024-03-26T02:48:09.276 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading mon daemons", 2024-03-26T02:48:09.276 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:48:09.276 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:48:09.504 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[88618]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.504 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[88618]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.504 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[88618]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.504 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[88618]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:09.505 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/1987690550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:09.505 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[83622]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.505 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[83622]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.505 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[83622]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.505 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[83622]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:09.505 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:09 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/1987690550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:09.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:09 smithi172 ceph-mon[67937]: from='client.54138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:09 smithi172 ceph-mon[67937]: from='client.54144 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:09 smithi172 ceph-mon[67937]: from='client.54150 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:09.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:09 smithi172 ceph-mon[67937]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:09.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:09 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/1987690550' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:10.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:10 smithi164 ceph-mon[83622]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:10.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:10 smithi164 ceph-mon[88618]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:10.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:10 smithi172 ceph-mon[67937]: from='client.54162 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:11.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:11 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:11.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:11 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:11.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:11 smithi164 ceph-mon[83622]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:11.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:11 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:11.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:11 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:11.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:11 smithi164 ceph-mon[88618]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:11.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:11 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:11.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:11 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:11.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:11 smithi172 ceph-mon[67937]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:12.046 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:48:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:48:12] "GET /metrics HTTP/1.1" 200 38015 "" "Prometheus/2.43.0" 2024-03-26T02:48:12.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:12 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:12.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:12 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:12.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:12 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:12.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:12 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:12 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:13.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:12 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:13.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:13 smithi164 ceph-mon[83622]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:13.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:13 smithi164 ceph-mon[88618]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:13 smithi172 ceph-mon[67937]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:16.003 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:16.003 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:16.003 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.003 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.003 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:16.003 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:16.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:16.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:16.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:16.004 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:16.004 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:16.004 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.004 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:16.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:16.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:16.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:16.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:16.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:16.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:16.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:16.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:16.983 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:16 smithi164 ceph-mon[83622]: Reconfiguring mon.a (monmap changed)... 2024-03-26T02:48:16.984 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:16 smithi164 ceph-mon[83622]: Reconfiguring daemon mon.a on smithi164 2024-03-26T02:48:16.984 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:16 smithi164 ceph-mon[88618]: Reconfiguring mon.a (monmap changed)... 2024-03-26T02:48:16.984 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:16 smithi164 ceph-mon[88618]: Reconfiguring daemon mon.a on smithi164 2024-03-26T02:48:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:16 smithi172 ceph-mon[67937]: Reconfiguring mon.a (monmap changed)... 2024-03-26T02:48:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:16 smithi172 ceph-mon[67937]: Reconfiguring daemon mon.a on smithi164 2024-03-26T02:48:17.980 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[88618]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:17.980 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:17.980 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:17.980 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:48:17.980 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:48:17.981 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:17.981 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[83622]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:17.981 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:17.981 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:17.981 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:48:17.981 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:48:17.981 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:17 smithi172 ceph-mon[67937]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:48:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:48:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[83622]: Reconfiguring mgr.y (monmap changed)... 2024-03-26T02:48:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[83622]: Reconfiguring daemon mgr.y on smithi164 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[88618]: Reconfiguring mgr.y (monmap changed)... 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[88618]: Reconfiguring daemon mgr.y on smithi164 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:19.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:19.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:19.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:19.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:18 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:18 smithi172 ceph-mon[67937]: Reconfiguring mgr.y (monmap changed)... 2024-03-26T02:48:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:18 smithi172 ceph-mon[67937]: Reconfiguring daemon mgr.y on smithi164 2024-03-26T02:48:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:18 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:18 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:18 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:18 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:18 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:18 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:19 smithi164 ceph-mon[83622]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:48:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:19 smithi164 ceph-mon[83622]: Reconfiguring daemon mon.c on smithi164 2024-03-26T02:48:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:19 smithi164 ceph-mon[83622]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:20.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:19 smithi164 ceph-mon[88618]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:48:20.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:19 smithi164 ceph-mon[88618]: Reconfiguring daemon mon.c on smithi164 2024-03-26T02:48:20.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:19 smithi164 ceph-mon[88618]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:19 smithi172 ceph-mon[67937]: Reconfiguring mon.c (monmap changed)... 2024-03-26T02:48:20.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:19 smithi172 ceph-mon[67937]: Reconfiguring daemon mon.c on smithi164 2024-03-26T02:48:20.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:19 smithi172 ceph-mon[67937]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[83622]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[83622]: Reconfiguring daemon osd.0 on smithi164 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[83622]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[88618]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:21.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[88618]: Reconfiguring daemon osd.0 on smithi164 2024-03-26T02:48:21.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:20 smithi164 ceph-mon[88618]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:20 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:20 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:20 smithi172 ceph-mon[67937]: Reconfiguring osd.0 (monmap changed)... 2024-03-26T02:48:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:20 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:48:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:20 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:20 smithi172 ceph-mon[67937]: Reconfiguring daemon osd.0 on smithi164 2024-03-26T02:48:21.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:20 smithi172 ceph-mon[67937]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:48:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:48:21] "GET /metrics HTTP/1.1" 200 38027 "" "Prometheus/2.43.0" 2024-03-26T02:48:22.565 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[88618]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[88618]: Reconfiguring daemon osd.1 on smithi164 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[83622]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:48:22.566 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:22.567 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:22 smithi164 ceph-mon[83622]: Reconfiguring daemon osd.1 on smithi164 2024-03-26T02:48:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:22 smithi172 ceph-mon[67937]: Reconfiguring osd.1 (monmap changed)... 2024-03-26T02:48:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:48:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:22 smithi172 ceph-mon[67937]: Reconfiguring daemon osd.1 on smithi164 2024-03-26T02:48:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:23 smithi172 ceph-mon[67937]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:23 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:23 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:23 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:48:23.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:23 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:23.828 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[88618]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:23.828 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:23.828 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:23.828 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:48:23.828 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:23.829 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[83622]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:23.829 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:23.829 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:23.829 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:48:23.829 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:23 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:24.776 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:24 smithi164 ceph-mon[83622]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T02:48:24.776 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:24 smithi164 ceph-mon[83622]: Reconfiguring daemon osd.2 on smithi164 2024-03-26T02:48:24.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:24 smithi164 ceph-mon[88618]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T02:48:24.776 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:24 smithi164 ceph-mon[88618]: Reconfiguring daemon osd.2 on smithi164 2024-03-26T02:48:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:24 smithi172 ceph-mon[67937]: Reconfiguring osd.2 (monmap changed)... 2024-03-26T02:48:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:24 smithi172 ceph-mon[67937]: Reconfiguring daemon osd.2 on smithi164 2024-03-26T02:48:25.788 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[88618]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:25.788 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:25.788 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:25.788 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:48:25.788 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:25.789 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[83622]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:25.789 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:25.789 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:25.789 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:48:25.789 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:25 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:25 smithi172 ceph-mon[67937]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:25 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:25 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:25 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:48:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:25 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[83622]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T02:48:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[83622]: Reconfiguring daemon osd.3 on smithi164 2024-03-26T02:48:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:26.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:48:26.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[88618]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T02:48:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[88618]: Reconfiguring daemon osd.3 on smithi164 2024-03-26T02:48:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:48:26.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:26 smithi172 ceph-mon[67937]: Reconfiguring osd.3 (monmap changed)... 2024-03-26T02:48:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:26 smithi172 ceph-mon[67937]: Reconfiguring daemon osd.3 on smithi164 2024-03-26T02:48:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:48:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:27.949 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:27 smithi164 ceph-mon[88618]: Reconfiguring rgw.foo.smithi164.cdxqhg (monmap changed)... 2024-03-26T02:48:27.949 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:27 smithi164 ceph-mon[88618]: Reconfiguring daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:48:27.949 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:27 smithi164 ceph-mon[88618]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:27.950 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:27 smithi164 ceph-mon[83622]: Reconfiguring rgw.foo.smithi164.cdxqhg (monmap changed)... 2024-03-26T02:48:27.950 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:27 smithi164 ceph-mon[83622]: Reconfiguring daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:48:27.950 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:27 smithi164 ceph-mon[83622]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:27 smithi172 ceph-mon[67937]: Reconfiguring rgw.foo.smithi164.cdxqhg (monmap changed)... 2024-03-26T02:48:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:27 smithi172 ceph-mon[67937]: Reconfiguring daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:48:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:27 smithi172 ceph-mon[67937]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:28.922 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:28 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:28.922 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:28 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:28.922 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:28 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:28.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:28 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:28.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:28 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:28.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:28 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:29.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T02:48:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T02:48:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:28 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:29.928 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:29 smithi172 ceph-mon[67937]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:48:29.928 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:29 smithi172 ceph-mon[67937]: Reconfiguring daemon mon.b on smithi172 2024-03-26T02:48:29.928 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:29 smithi172 ceph-mon[67937]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:29 smithi164 ceph-mon[83622]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:48:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:29 smithi164 ceph-mon[83622]: Reconfiguring daemon mon.b on smithi172 2024-03-26T02:48:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:29 smithi164 ceph-mon[83622]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:30.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:29 smithi164 ceph-mon[88618]: Reconfiguring mon.b (monmap changed)... 2024-03-26T02:48:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:29 smithi164 ceph-mon[88618]: Reconfiguring daemon mon.b on smithi172 2024-03-26T02:48:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:29 smithi164 ceph-mon[88618]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:48:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:48:30.785 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:48:31.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:48:31.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T02:48:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T02:48:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:31.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:31 smithi172 ceph-mon[67937]: Reconfiguring mgr.x (monmap changed)... 2024-03-26T02:48:31.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:31 smithi172 ceph-mon[67937]: Reconfiguring daemon mgr.x on smithi172 2024-03-26T02:48:31.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:31 smithi172 ceph-mon[67937]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:31.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:31.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:31.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:48:31.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[88618]: Reconfiguring mgr.x (monmap changed)... 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[88618]: Reconfiguring daemon mgr.x on smithi172 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[88618]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[83622]: Reconfiguring mgr.x (monmap changed)... 2024-03-26T02:48:32.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[83622]: Reconfiguring daemon mgr.x on smithi172 2024-03-26T02:48:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[83622]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:48:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:48:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:48:31] "GET /metrics HTTP/1.1" 200 38028 "" "Prometheus/2.43.0" 2024-03-26T02:48:32.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:32 smithi172 ceph-mon[67937]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T02:48:32.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:32 smithi172 ceph-mon[67937]: Reconfiguring daemon osd.4 on smithi172 2024-03-26T02:48:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:32 smithi164 ceph-mon[83622]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T02:48:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:32 smithi164 ceph-mon[83622]: Reconfiguring daemon osd.4 on smithi172 2024-03-26T02:48:33.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:32 smithi164 ceph-mon[88618]: Reconfiguring osd.4 (monmap changed)... 2024-03-26T02:48:33.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:32 smithi164 ceph-mon[88618]: Reconfiguring daemon osd.4 on smithi172 2024-03-26T02:48:33.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:33 smithi172 ceph-mon[67937]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:33.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:33 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:33.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:33 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:33.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:33 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:48:33.895 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:33 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[83622]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:34.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:34.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:48:34.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[88618]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:48:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:33 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:34 smithi164 ceph-mon[83622]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T02:48:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:34 smithi164 ceph-mon[83622]: Reconfiguring daemon osd.5 on smithi172 2024-03-26T02:48:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:34 smithi164 ceph-mon[88618]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T02:48:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:34 smithi164 ceph-mon[88618]: Reconfiguring daemon osd.5 on smithi172 2024-03-26T02:48:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:34 smithi172 ceph-mon[67937]: Reconfiguring osd.5 (monmap changed)... 2024-03-26T02:48:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:34 smithi172 ceph-mon[67937]: Reconfiguring daemon osd.5 on smithi172 2024-03-26T02:48:35.714 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:35 smithi172 ceph-mon[67937]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:35.714 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:35.714 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:35.714 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:48:35.715 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[83622]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:36.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:48:36.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[88618]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:48:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:36.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:36 smithi172 ceph-mon[67937]: Reconfiguring osd.6 (monmap changed)... 2024-03-26T02:48:36.961 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:36 smithi172 ceph-mon[67937]: Reconfiguring daemon osd.6 on smithi172 2024-03-26T02:48:36.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:36 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:36.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:36 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:36.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:36 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:48:36.962 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:36 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:37.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[83622]: Reconfiguring osd.6 (monmap changed)... 2024-03-26T02:48:37.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[83622]: Reconfiguring daemon osd.6 on smithi172 2024-03-26T02:48:37.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:37.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:37.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:48:37.004 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:37.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[88618]: Reconfiguring osd.6 (monmap changed)... 2024-03-26T02:48:37.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[88618]: Reconfiguring daemon osd.6 on smithi172 2024-03-26T02:48:37.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:37.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:37.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:48:37.005 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:36 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:37.891 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:37 smithi172 ceph-mon[67937]: Reconfiguring osd.7 (monmap changed)... 2024-03-26T02:48:37.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:37 smithi172 ceph-mon[67937]: Reconfiguring daemon osd.7 on smithi172 2024-03-26T02:48:37.892 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:37 smithi172 ceph-mon[67937]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:38.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:37 smithi164 ceph-mon[88618]: Reconfiguring osd.7 (monmap changed)... 2024-03-26T02:48:38.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:37 smithi164 ceph-mon[88618]: Reconfiguring daemon osd.7 on smithi172 2024-03-26T02:48:38.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:37 smithi164 ceph-mon[88618]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:37 smithi164 ceph-mon[83622]: Reconfiguring osd.7 (monmap changed)... 2024-03-26T02:48:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:37 smithi164 ceph-mon[83622]: Reconfiguring daemon osd.7 on smithi172 2024-03-26T02:48:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:37 smithi164 ceph-mon[83622]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:38.886 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:38 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:38.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:38 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:38.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:38 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:38.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:38 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:48:38.887 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:38 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:48:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:48:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:38 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:39.660 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:48:39.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: Reconfiguring rgw.foo.smithi172.gbbkmx (monmap changed)... 2024-03-26T02:48:39.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: Reconfiguring daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:48:39.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:39.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:39.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:39.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: Reconfiguring rgw.foo.smithi172.gbbkmx (monmap changed)... 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: Reconfiguring daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-26T02:48:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: Reconfiguring rgw.foo.smithi172.gbbkmx (monmap changed)... 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: Reconfiguring daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:40.341 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:48:40.341 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (3m) 29s ago 25m 17.7M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:48:40.341 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (3m) 51s ago 24m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (4m) 29s ago 24m 52.4M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (4m) 51s ago 29m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (14m) 29s ago 31m 497M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (66s) 29s ago 31m 37.6M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (116s) 51s ago 30m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (40s) 29s ago 30m 20.7M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (3m) 29s ago 25m 7982k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (3m) 51s ago 25m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (29m) 29s ago 29m 55.1M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (29m) 29s ago 29m 56.8M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:48:40.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (28m) 29s ago 28m 53.6M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:48:40.343 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (28m) 29s ago 28m 57.7M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:48:40.343 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (27m) 51s ago 27m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:48:40.343 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (27m) 51s ago 27m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:48:40.343 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (27m) 51s ago 27m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:48:40.343 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (26m) 51s ago 26m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:48:40.343 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (3m) 51s ago 26m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:48:40.343 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (24m) 29s ago 24m 78.1M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:48:40.343 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (24m) 51s ago 24m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:48:40.743 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:48:40.744 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:48:40.745 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-26T02:48:40.745 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 5 2024-03-26T02:48:40.745 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:48:40.745 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mon 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all crash 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mds 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:48:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all nfs 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all node-exporter 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all prometheus 2024-03-26T02:48:41.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all alertmanager 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mon 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all crash 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mds 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all nfs 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all node-exporter 2024-03-26T02:48:41.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all prometheus 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all alertmanager 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all grafana 2024-03-26T02:48:41.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all loki 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all promtail 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Finalizing container_image settings 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:48:41.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all grafana 2024-03-26T02:48:41.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all loki 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all promtail 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Finalizing container_image settings 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:48:41.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:48:41.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: Upgrade: Complete! 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.055 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.056 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: Upgrade: Complete! 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:41.057 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mon 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all crash 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mds 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='client.54168 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nfs 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all node-exporter 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all prometheus 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all alertmanager 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all grafana 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all loki 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:41.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all promtail 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Finalizing container_image settings 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:48:41.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: Upgrade: Complete! 2024-03-26T02:48:41.062 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:48:41.063 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:41.079 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:48:41.079 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": null, 2024-03-26T02:48:41.079 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": false, 2024-03-26T02:48:41.079 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "", 2024-03-26T02:48:41.079 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:48:41.079 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": null, 2024-03-26T02:48:41.079 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:48:41.080 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:48:41.080 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:48:42.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:41 smithi164 ceph-mon[83622]: from='client.54174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:42.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:41 smithi164 ceph-mon[83622]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:42.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:41 smithi164 ceph-mon[83622]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:42.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:41 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/440084331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:42.002 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:41 smithi164 ceph-mon[88618]: from='client.54174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:42.002 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:41 smithi164 ceph-mon[88618]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:42.002 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:41 smithi164 ceph-mon[88618]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:42.002 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:41 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/440084331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:42.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:41 smithi172 ceph-mon[67937]: from='client.54174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:41 smithi172 ceph-mon[67937]: from='client.54180 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:41 smithi172 ceph-mon[67937]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:41 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/440084331' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:48:42.291 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:48:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:48:42] "GET /metrics HTTP/1.1" 200 38028 "" "Prometheus/2.43.0" 2024-03-26T02:48:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:42 smithi164 ceph-mon[83622]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:42 smithi164 ceph-mon[88618]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:42 smithi172 ceph-mon[67937]: from='client.64126 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:48:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:43 smithi164 ceph-mon[83622]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:44.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:43 smithi164 ceph-mon[88618]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:43 smithi172 ceph-mon[67937]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:45 smithi164 ceph-mon[83622]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:45 smithi164 ceph-mon[88618]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:45 smithi172 ceph-mon[67937]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:48:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:48:48.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:47 smithi164 ceph-mon[83622]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:47 smithi164 ceph-mon[88618]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:48.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:47 smithi172 ceph-mon[67937]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:48 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:48 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:48 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:49 smithi164 ceph-mon[83622]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:49 smithi164 ceph-mon[88618]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:49 smithi172 ceph-mon[67937]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:52.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:51 smithi164 ceph-mon[83622]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:52.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:51 smithi164 ceph-mon[88618]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:52.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:51 smithi172 ceph-mon[67937]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:48:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:48:52] "GET /metrics HTTP/1.1" 200 38027 "" "Prometheus/2.43.0" 2024-03-26T02:48:54.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:53 smithi164 ceph-mon[83622]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:53 smithi164 ceph-mon[88618]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:54.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:53 smithi172 ceph-mon[67937]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:56.011 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:55 smithi172 ceph-mon[67937]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:55 smithi164 ceph-mon[83622]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:55 smithi164 ceph-mon[88618]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:48:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:57 smithi164 ceph-mon[83622]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:58.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:57 smithi164 ceph-mon[88618]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:58.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:57 smithi172 ceph-mon[67937]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:48:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:58 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:59.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:58 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:48:59.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:58 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:00.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:48:59 smithi164 ceph-mon[83622]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:00.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:48:59 smithi164 ceph-mon[88618]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:00.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:48:59 smithi172 ceph-mon[67937]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:01.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:01.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:01.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:02.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:01 smithi164 ceph-mon[83622]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:02.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:01 smithi164 ceph-mon[88618]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:01 smithi172 ceph-mon[67937]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:49:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:49:01] "GET /metrics HTTP/1.1" 200 38023 "" "Prometheus/2.43.0" 2024-03-26T02:49:04.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:03 smithi164 ceph-mon[83622]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:04.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:03 smithi164 ceph-mon[88618]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:04.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:03 smithi172 ceph-mon[67937]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:05.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:05 smithi172 ceph-mon[67937]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:05 smithi164 ceph-mon[83622]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:05 smithi164 ceph-mon[88618]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:08.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:07 smithi164 ceph-mon[83622]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:08.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:07 smithi164 ceph-mon[88618]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:07 smithi172 ceph-mon[67937]: pgmap v44: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:08 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:09.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:08 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:08 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:10.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:09 smithi164 ceph-mon[83622]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:10.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:09 smithi164 ceph-mon[88618]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:10.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:09 smithi172 ceph-mon[67937]: pgmap v45: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:12.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:11 smithi164 ceph-mon[83622]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:12.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:11 smithi164 ceph-mon[88618]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:12.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:11 smithi172 ceph-mon[67937]: pgmap v46: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:12.249 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:49:12.277 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:49:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:49:11] "GET /metrics HTTP/1.1" 200 38023 "" "Prometheus/2.43.0" 2024-03-26T02:49:12.896 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:12 smithi164 ceph-mon[83622]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:12.896 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:12 smithi164 ceph-mon[88618]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:12 smithi172 ceph-mon[67937]: from='client.54198 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:13.836 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:13 smithi164 ceph-mon[83622]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:13.836 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:13 smithi164 ceph-mon[88618]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:13 smithi172 ceph-mon[67937]: pgmap v47: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:49:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (3m) 63s ago 25m 17.7M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:49:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (3m) 85s ago 25m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:49:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (4m) 63s ago 24m 52.4M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:49:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (4m) 85s ago 30m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:49:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (15m) 63s ago 31m 497M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:49:14.341 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (100s) 63s ago 32m 37.6M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (2m) 85s ago 30m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (74s) 63s ago 30m 20.7M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (4m) 63s ago 26m 7982k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (4m) 85s ago 26m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (30m) 63s ago 30m 55.1M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (29m) 63s ago 29m 56.8M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (29m) 63s ago 29m 53.6M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (28m) 63s ago 28m 57.7M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (28m) 85s ago 28m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (28m) 85s ago 28m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (27m) 85s ago 27m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (27m) 85s ago 27m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:49:14.342 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (4m) 85s ago 26m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:49:14.343 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (25m) 63s ago 25m 78.1M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:49:14.343 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (25m) 85s ago 25m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:49:15.017 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:14 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:15.018 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:14 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:14 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:15.085 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.mon | length == 1'"'"'' 2024-03-26T02:49:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:15 smithi164 ceph-mon[83622]: from='client.64138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:15 smithi164 ceph-mon[83622]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:16.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:15 smithi164 ceph-mon[88618]: from='client.64138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:16.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:15 smithi164 ceph-mon[88618]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:15 smithi172 ceph-mon[67937]: from='client.64138 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:15 smithi172 ceph-mon[67937]: pgmap v48: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:17.232 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:49:17.842 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:17 smithi164 ceph-mon[83622]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:17.843 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:17 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/1249777749' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:17.843 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:17 smithi164 ceph-mon[88618]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:17.843 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:17 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/1249777749' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:17.867 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.mon | keys'"'"' | grep $sha1' 2024-03-26T02:49:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:17 smithi172 ceph-mon[67937]: pgmap v49: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:17 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/1249777749' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:18 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:18 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:18 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:19.926 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)" 2024-03-26T02:49:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:19 smithi164 ceph-mon[83622]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:20.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:19 smithi164 ceph-mon[88618]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:19 smithi172 ceph-mon[67937]: pgmap v50: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:20.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 5'"'"'' 2024-03-26T02:49:20.892 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:20 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/4074337727' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:20.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:20 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/4074337727' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:20 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/4074337727' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:22.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:21 smithi164 ceph-mon[88618]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:22.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:21 smithi164 ceph-mon[83622]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:22.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:49:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:49:21] "GET /metrics HTTP/1.1" 200 38025 "" "Prometheus/2.43.0" 2024-03-26T02:49:22.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:21 smithi172 ceph-mon[67937]: pgmap v51: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:23.918 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:23 smithi164 ceph-mon[88618]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:23.918 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:23 smithi164 ceph-mon[88618]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:23.918 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:23 smithi164 ceph-mon[83622]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:23.918 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:23 smithi164 ceph-mon[83622]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:24.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:23 smithi172 ceph-mon[67937]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:24.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:23 smithi172 ceph-mon[67937]: from='client.64156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:24.842 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:49:25.480 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade status' 2024-03-26T02:49:26.012 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:25 smithi164 ceph-mon[83622]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:26.012 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:25 smithi164 ceph-mon[88618]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:25 smithi172 ceph-mon[67937]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": null, 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": false, 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "", 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": null, 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:49:27.580 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:49:27.904 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:27 smithi164 ceph-mon[83622]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:27.904 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:27 smithi164 ceph-mon[88618]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:27 smithi172 ceph-mon[67937]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:28.203 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:49:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:28 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:28 smithi164 ceph-mon[83622]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:28 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:28 smithi164 ceph-mon[88618]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:29.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:28 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:28 smithi172 ceph-mon[67937]: from='client.54228 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:29 smithi164 ceph-mon[83622]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:29 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:30.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:29 smithi164 ceph-mon[88618]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:30.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:29 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:30.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:29 smithi172 ceph-mon[67937]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:30.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:29 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:30.183 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:49:30.843 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:30 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/1761189631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:49:30.843 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:30 smithi164 ceph-mon[83622]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:30.843 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:30 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/1761189631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:49:30.844 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:30 smithi164 ceph-mon[88618]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:30.867 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types osd --limit 2' 2024-03-26T02:49:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:30 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/1761189631' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:49:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:30 smithi172 ceph-mon[67937]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:32.167 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:49:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:49:31] "GET /metrics HTTP/1.1" 200 38026 "" "Prometheus/2.43.0" 2024-03-26T02:49:34.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:33 smithi164 ceph-mon[88618]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:33 smithi164 ceph-mon[83622]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:34.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:33 smithi172 ceph-mon[67937]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:34.922 INFO:teuthology.orchestra.run.smithi164.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:49:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:34 smithi164 ceph-mon[83622]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:35.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:34 smithi164 ceph-mon[88618]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:34 smithi172 ceph-mon[67937]: from='client.64174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "osd", "limit": 2, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:35.611 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-26T02:49:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[83622]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.937 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.937 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[88618]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:35.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:35.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:35.938 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:35 smithi172 ceph-mon[67937]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:37.033 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:36 smithi164 ceph-mon[83622]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:49:37.034 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:36 smithi164 ceph-mon[88618]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:49:37.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:36 smithi172 ceph-mon[67937]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:49:37.874 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:49:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:37 smithi164 ceph-mon[83622]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:37 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:49:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:37 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:49:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:37 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:37 smithi164 ceph-mon[88618]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:37 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:49:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:37 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:49:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:37 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:37 smithi172 ceph-mon[67937]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:37 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:49:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:37 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:49:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:37 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (4m) 88s ago 26m 17.7M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (4m) 109s ago 25m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (5m) 88s ago 25m 52.4M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (5m) 109s ago 30m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (15m) 88s ago 32m 497M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (2m) 88s ago 32m 37.6M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (2m) 109s ago 30m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (98s) 88s ago 31m 20.7M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (4m) 88s ago 26m 7982k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (4m) 109s ago 26m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (30m) 88s ago 30m 55.1M 2992M 17.2.0 e1d6a67b021e 66c629085604 2024-03-26T02:49:38.557 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (30m) 88s ago 30m 56.8M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (29m) 88s ago 29m 53.6M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (29m) 88s ago 29m 57.7M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (28m) 109s ago 28m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (28m) 109s ago 28m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (28m) 109s ago 28m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (27m) 109s ago 27m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (4m) 109s ago 27m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (25m) 88s ago 25m 78.1M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:49:38.558 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (25m) 109s ago 25m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:49:38.969 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:49:38.970 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:49:38.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:49:38.971 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:49:38.971 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-03-26T02:49:38.971 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 5 2024-03-26T02:49:38.971 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:49:38.971 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:49:38.971 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:38 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:38.971 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:38 smithi164 ceph-mon[83622]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:49:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:38 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:38.972 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:38 smithi164 ceph-mon[88618]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:49:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:38 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:38 smithi172 ceph-mon[67937]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (2 remaining).", 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "", 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 image", 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:49:39.308 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:49:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='client.54246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/3293210212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='client.54246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/3293210212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='client.54246 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='client.54252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='client.64192 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/3293210212' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:49:40.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:40.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:49:41.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[88618]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:41.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[88618]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[88618]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mgr 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mon 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all crash 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[88618]: Upgrade: osd.0 is safe to restart 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[83622]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[83622]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[83622]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mgr 2024-03-26T02:49:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mon 2024-03-26T02:49:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all crash 2024-03-26T02:49:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:49:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:40 smithi164 ceph-mon[83622]: Upgrade: osd.0 is safe to restart 2024-03-26T02:49:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:40 smithi172 ceph-mon[67937]: from='client.54270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:49:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:40 smithi172 ceph-mon[67937]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:49:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:40 smithi172 ceph-mon[67937]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:49:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mgr 2024-03-26T02:49:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mon 2024-03-26T02:49:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all crash 2024-03-26T02:49:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:40 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-03-26T02:49:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:40 smithi172 ceph-mon[67937]: Upgrade: osd.0 is safe to restart 2024-03-26T02:49:42.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:49:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:49:42] "GET /metrics HTTP/1.1" 200 38026 "" "Prometheus/2.43.0" 2024-03-26T02:49:42.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:41 smithi164 ceph-mon[83622]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:49:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:49:42.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:41 smithi164 ceph-mon[88618]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:42.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:42.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:49:42.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:49:42.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:41 smithi172 ceph-mon[67937]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:42.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T02:49:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:49:43.003 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:42 smithi164 ceph-mon[83622]: Upgrade: Updating osd.0 2024-03-26T02:49:43.003 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:42 smithi164 ceph-mon[83622]: Deploying daemon osd.0 on smithi164 2024-03-26T02:49:43.003 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:42 smithi164 ceph-mon[88618]: Upgrade: Updating osd.0 2024-03-26T02:49:43.003 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:42 smithi164 ceph-mon[88618]: Deploying daemon osd.0 on smithi164 2024-03-26T02:49:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:42 smithi172 ceph-mon[67937]: Upgrade: Updating osd.0 2024-03-26T02:49:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:42 smithi172 ceph-mon[67937]: Deploying daemon osd.0 on smithi164 2024-03-26T02:49:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:43 smithi164 ceph-mon[83622]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:44.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:43 smithi164 ceph-mon[88618]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:43 smithi172 ceph-mon[67937]: pgmap v62: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:44.784 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:44 smithi164 systemd[1]: Stopping Ceph osd.0 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:49:45.044 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[32388]: 2024-03-26T02:49:44.782+0000 7fbccc84c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:49:45.044 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[32388]: 2024-03-26T02:49:44.782+0000 7fbccc84c700 -1 osd.0 112 *** Got signal Terminated *** 2024-03-26T02:49:45.044 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[32388]: 2024-03-26T02:49:44.782+0000 7fbccc84c700 -1 osd.0 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:49:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:45 smithi164 ceph-mon[88618]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:45 smithi164 ceph-mon[88618]: osd.0 marked itself down and dead 2024-03-26T02:49:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:45 smithi164 ceph-mon[83622]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:45 smithi164 ceph-mon[83622]: osd.0 marked itself down and dead 2024-03-26T02:49:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:45 smithi172 ceph-mon[67937]: pgmap v63: 161 pgs: 161 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:49:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:49:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:45 smithi172 ceph-mon[67937]: osd.0 marked itself down and dead 2024-03-26T02:49:46.441 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:46 smithi164 podman[95540]: 2024-03-26 02:49:46.154474391 +0000 UTC m=+1.487236556 container died 66c6290856043f68d96075ebde6c2f618bf0b39b806ce3197da31eaa670e5b57 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True) 2024-03-26T02:49:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:46 smithi164 ceph-mon[83622]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:49:46.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:46 smithi164 ceph-mon[83622]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T02:49:46.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:46 smithi164 ceph-mon[88618]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:49:46.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:46 smithi164 ceph-mon[88618]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T02:49:46.795 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:46 smithi164 podman[95540]: 2024-03-26 02:49:46.441194727 +0000 UTC m=+1.773956900 container cleanup 66c6290856043f68d96075ebde6c2f618bf0b39b806ce3197da31eaa670e5b57 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, distribution-scope=public, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-26T02:49:46.795 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:46 smithi164 bash[95540]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0 2024-03-26T02:49:46.795 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:46 smithi164 podman[95557]: 2024-03-26 02:49:46.684003379 +0000 UTC m=+0.526409436 container remove 66c6290856043f68d96075ebde6c2f618bf0b39b806ce3197da31eaa670e5b57 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, version=8, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.openshift.expose-services=, name=centos-stream) 2024-03-26T02:49:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:46 smithi172 ceph-mon[67937]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:49:47.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:46 smithi172 ceph-mon[67937]: osdmap e113: 8 total, 7 up, 8 in 2024-03-26T02:49:47.294 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:47 smithi164 podman[95647]: 2024-03-26 02:49:46.914970925 +0000 UTC m=+0.018202962 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:49:47.294 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:47 smithi164 podman[95647]: 2024-03-26 02:49:47.035297983 +0000 UTC m=+0.138530021 container create f214878aeb6be02f0df8454a2282c191859820b2207b67a8ff07ebf3d46d627e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:49:47.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:47 smithi164 podman[95647]: 2024-03-26 02:49:47.254950931 +0000 UTC m=+0.358182970 container init f214878aeb6be02f0df8454a2282c191859820b2207b67a8ff07ebf3d46d627e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:49:47.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:47 smithi164 podman[95647]: 2024-03-26 02:49:47.258214368 +0000 UTC m=+0.361446407 container start f214878aeb6be02f0df8454a2282c191859820b2207b67a8ff07ebf3d46d627e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T02:49:47.557 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:47 smithi164 podman[95647]: 2024-03-26 02:49:47.337937879 +0000 UTC m=+0.441169928 container attach f214878aeb6be02f0df8454a2282c191859820b2207b67a8ff07ebf3d46d627e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-26T02:49:47.916 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:47 smithi164 ceph-mon[83622]: pgmap v65: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:49:47.916 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:47 smithi164 ceph-mon[83622]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T02:49:47.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:47 smithi164 ceph-mon[88618]: pgmap v65: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:49:47.917 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:47 smithi164 ceph-mon[88618]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T02:49:47.917 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:47 smithi164 podman[95647]: 2024-03-26 02:49:47.557686655 +0000 UTC m=+0.660918699 container died f214878aeb6be02f0df8454a2282c191859820b2207b67a8ff07ebf3d46d627e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:49:48.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:47 smithi172 ceph-mon[67937]: pgmap v65: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:49:48.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:47 smithi172 ceph-mon[67937]: osdmap e114: 8 total, 7 up, 8 in 2024-03-26T02:49:48.294 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 podman[95673]: 2024-03-26 02:49:48.108422538 +0000 UTC m=+0.545151301 container remove f214878aeb6be02f0df8454a2282c191859820b2207b67a8ff07ebf3d46d627e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T02:49:48.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-26T02:49:48.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 systemd[1]: Stopped Ceph osd.0 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:49:48.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.0.service: Consumed 14.015s CPU time. 2024-03-26T02:49:48.683 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 systemd[1]: Starting Ceph osd.0 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:49:48.683 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 podman[95780]: 2024-03-26 02:49:48.457308105 +0000 UTC m=+0.018203669 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:49:48.683 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 podman[95780]: 2024-03-26 02:49:48.57648467 +0000 UTC m=+0.137380206 container create 2a93d82160d3c47142d856d4a67dd455312fd943b4fe7530c2cdb158e67d4a3b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T02:49:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:48 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:48 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:49.045 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 podman[95780]: 2024-03-26 02:49:48.784104295 +0000 UTC m=+0.344999847 container init 2a93d82160d3c47142d856d4a67dd455312fd943b4fe7530c2cdb158e67d4a3b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:49:49.045 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 podman[95780]: 2024-03-26 02:49:48.787776006 +0000 UTC m=+0.348671552 container start 2a93d82160d3c47142d856d4a67dd455312fd943b4fe7530c2cdb158e67d4a3b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T02:49:49.045 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:48 smithi164 podman[95780]: 2024-03-26 02:49:48.866949181 +0000 UTC m=+0.427844732 container attach 2a93d82160d3c47142d856d4a67dd455312fd943b4fe7530c2cdb158e67d4a3b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:49:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:48 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:49.544 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate[95794]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T02:49:49.544 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate[95794]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 bash[95780]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 bash[95780]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 bash[95780]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 bash[95780]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 bash[95780]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 bash[95780]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 bash[95780]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate[95794]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate[95794]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate[95794]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate[95794]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate[95794]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-26T02:49:49.545 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 podman[95780]: 2024-03-26 02:49:49.256207276 +0000 UTC m=+0.817102814 container died 2a93d82160d3c47142d856d4a67dd455312fd943b4fe7530c2cdb158e67d4a3b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T02:49:49.912 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-mon[83622]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:49.913 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:49 smithi164 ceph-mon[88618]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:49 smithi172 ceph-mon[67937]: pgmap v67: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:49:50.265 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:49 smithi164 podman[95915]: 2024-03-26 02:49:49.912801043 +0000 UTC m=+0.651878354 container remove 2a93d82160d3c47142d856d4a67dd455312fd943b4fe7530c2cdb158e67d4a3b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-activate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T02:49:50.265 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:50 smithi164 podman[95983]: 2024-03-26 02:49:50.028289442 +0000 UTC m=+0.018188064 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:49:50.265 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:50 smithi164 podman[95983]: 2024-03-26 02:49:50.138835434 +0000 UTC m=+0.128734034 container create 1b92fd14a23a69fbb4f037226ac3bcda32850bd0dfdbe8fd389fd822f55a1fa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:49:50.794 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:50 smithi164 podman[95983]: 2024-03-26 02:49:50.333169457 +0000 UTC m=+0.323068070 container init 1b92fd14a23a69fbb4f037226ac3bcda32850bd0dfdbe8fd389fd822f55a1fa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T02:49:50.794 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:50 smithi164 podman[95983]: 2024-03-26 02:49:50.336360986 +0000 UTC m=+0.326259587 container start 1b92fd14a23a69fbb4f037226ac3bcda32850bd0dfdbe8fd389fd822f55a1fa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T02:49:50.795 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:50 smithi164 ceph-osd[96002]: -- 172.21.15.164:0/385330475 <== mon.0 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d581acf680 con 0x55d581a99400 2024-03-26T02:49:50.795 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:50 smithi164 bash[95983]: 1b92fd14a23a69fbb4f037226ac3bcda32850bd0dfdbe8fd389fd822f55a1fa0 2024-03-26T02:49:50.795 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:50 smithi164 systemd[1]: Started Ceph osd.0 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:49:51.294 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:50 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[95998]: 2024-03-26T02:49:50.872+0000 7fddb2e1b7c0 -1 Falling back to public interface 2024-03-26T02:49:51.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:51.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[83622]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:49:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:51.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:51 smithi164 ceph-mon[88618]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:49:51.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:51 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:51 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:51 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:51 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:49:51.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:51 smithi172 ceph-mon[67937]: pgmap v68: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:49:52.287 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:49:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:49:51] "GET /metrics HTTP/1.1" 200 38021 "" "Prometheus/2.43.0" 2024-03-26T02:49:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:52 smithi164 ceph-mon[83622]: pgmap v69: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 72/723 objects degraded (9.959%) 2024-03-26T02:49:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:52 smithi164 ceph-mon[83622]: Health check failed: Degraded data redundancy: 72/723 objects degraded (9.959%), 18 pgs degraded (PG_DEGRADED) 2024-03-26T02:49:53.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:52 smithi164 ceph-mon[88618]: pgmap v69: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 72/723 objects degraded (9.959%) 2024-03-26T02:49:53.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:52 smithi164 ceph-mon[88618]: Health check failed: Degraded data redundancy: 72/723 objects degraded (9.959%), 18 pgs degraded (PG_DEGRADED) 2024-03-26T02:49:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:52 smithi172 ceph-mon[67937]: pgmap v69: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 100 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 72/723 objects degraded (9.959%) 2024-03-26T02:49:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:52 smithi172 ceph-mon[67937]: Health check failed: Degraded data redundancy: 72/723 objects degraded (9.959%), 18 pgs degraded (PG_DEGRADED) 2024-03-26T02:49:53.873 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:53 smithi164 ceph-mon[88618]: from='osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:49:53.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:53 smithi164 ceph-mon[88618]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:49:53.874 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[95998]: 2024-03-26T02:49:53.586+0000 7fddb2e1b7c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-03-26T02:49:53.874 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[95998]: 2024-03-26T02:49:53.615+0000 7fddb2e1b7c0 -1 osd.0 112 log_to_monitors true 2024-03-26T02:49:53.874 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:53 smithi164 ceph-mon[83622]: from='osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:49:53.874 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:53 smithi164 ceph-mon[83622]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:49:54.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:53 smithi172 ceph-mon[67937]: from='osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:49:54.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:53 smithi172 ceph-mon[67937]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T02:49:54.542 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[95998]: 2024-03-26T02:49:54.244+0000 7fdda026b700 -1 osd.0 112 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:49:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[83622]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:49:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[83622]: osdmap e115: 8 total, 7 up, 8 in 2024-03-26T02:49:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[83622]: from='osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:49:54.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[83622]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:49:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[83622]: from='osd.0 ' entity='osd.0' 2024-03-26T02:49:54.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[83622]: pgmap v71: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 72/723 objects degraded (9.959%) 2024-03-26T02:49:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[88618]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:49:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[88618]: osdmap e115: 8 total, 7 up, 8 in 2024-03-26T02:49:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[88618]: from='osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:49:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[88618]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:49:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[88618]: from='osd.0 ' entity='osd.0' 2024-03-26T02:49:54.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:54 smithi164 ceph-mon[88618]: pgmap v71: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 72/723 objects degraded (9.959%) 2024-03-26T02:49:55.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:54 smithi172 ceph-mon[67937]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T02:49:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:54 smithi172 ceph-mon[67937]: osdmap e115: 8 total, 7 up, 8 in 2024-03-26T02:49:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:54 smithi172 ceph-mon[67937]: from='osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:49:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:54 smithi172 ceph-mon[67937]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:49:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:54 smithi172 ceph-mon[67937]: from='osd.0 ' entity='osd.0' 2024-03-26T02:49:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:54 smithi172 ceph-mon[67937]: pgmap v71: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 72/723 objects degraded (9.959%) 2024-03-26T02:49:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[83622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:49:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[83622]: osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125] boot 2024-03-26T02:49:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[83622]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T02:49:56.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:49:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[88618]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:49:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[88618]: osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125] boot 2024-03-26T02:49:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[88618]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T02:49:56.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:55 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:49:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:55 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:55 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:55 smithi172 ceph-mon[67937]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:49:56.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:55 smithi172 ceph-mon[67937]: osd.0 [v2:172.21.15.164:6802/2009859125,v1:172.21.15.164:6803/2009859125] boot 2024-03-26T02:49:56.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:55 smithi172 ceph-mon[67937]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T02:49:56.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:55 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:49:57.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:57.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:57.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:57 smithi164 ceph-mon[83622]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T02:49:57.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:57 smithi164 ceph-mon[83622]: pgmap v74: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-03-26T02:49:57.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:57.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:57.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:57 smithi164 ceph-mon[88618]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T02:49:57.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:57 smithi164 ceph-mon[88618]: pgmap v74: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-03-26T02:49:57.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:57.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:49:57.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:57 smithi172 ceph-mon[67937]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T02:49:57.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:57 smithi172 ceph-mon[67937]: pgmap v74: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-03-26T02:49:58.364 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:58 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:58.365 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:58 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:58.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:58 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:49:59.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:49:59 smithi164 ceph-mon[83622]: pgmap v75: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-03-26T02:49:59.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:49:59 smithi164 ceph-mon[88618]: pgmap v75: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-03-26T02:49:59.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:49:59 smithi172 ceph-mon[67937]: pgmap v75: 161 pgs: 37 peering, 7 active+undersized+degraded, 14 active+undersized, 103 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 32/723 objects degraded (4.426%) 2024-03-26T02:50:00.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:49:59 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:49:59.874+0000 7f6ddf0a2700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-26T02:50:00.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: Health check update: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T02:50:00.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: Health detail: HEALTH_WARN Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded 2024-03-26T02:50:00.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: [WRN] PG_DEGRADED: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[83622]: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: Health check update: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: Health detail: HEALTH_WARN Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: [WRN] PG_DEGRADED: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-03-26T02:50:00.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-03-26T02:50:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-03-26T02:50:00.548 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:00 smithi164 ceph-mon[88618]: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: Health check update: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: Health detail: HEALTH_WARN Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded 2024-03-26T02:50:00.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: [WRN] PG_DEGRADED: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded 2024-03-26T02:50:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-03-26T02:50:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-03-26T02:50:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-03-26T02:50:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-03-26T02:50:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-03-26T02:50:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-03-26T02:50:00.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:00 smithi172 ceph-mon[67937]: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-03-26T02:50:01.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:01 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:01.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:01 smithi164 ceph-mon[83622]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-26T02:50:01.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:01 smithi164 ceph-mon[83622]: pgmap v76: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-03-26T02:50:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:01 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:01 smithi164 ceph-mon[88618]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-26T02:50:01.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:01 smithi164 ceph-mon[88618]: pgmap v76: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-03-26T02:50:01.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:01 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:01.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:01 smithi172 ceph-mon[67937]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-26T02:50:01.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:01 smithi172 ceph-mon[67937]: pgmap v76: 161 pgs: 37 peering, 124 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-03-26T02:50:02.266 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:50:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:50:02] "GET /metrics HTTP/1.1" 200 38113 "" "Prometheus/2.43.0" 2024-03-26T02:50:02.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:02 smithi164 ceph-mon[88618]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-03-26T02:50:02.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:02 smithi164 ceph-mon[88618]: Cluster is now healthy 2024-03-26T02:50:02.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:02 smithi164 ceph-mon[83622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-03-26T02:50:02.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:02 smithi164 ceph-mon[83622]: Cluster is now healthy 2024-03-26T02:50:02.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:02 smithi172 ceph-mon[67937]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/723 objects degraded (4.426%), 7 pgs degraded) 2024-03-26T02:50:02.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:02 smithi172 ceph-mon[67937]: Cluster is now healthy 2024-03-26T02:50:03.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:03 smithi164 ceph-mon[83622]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:03.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:03 smithi164 ceph-mon[88618]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:03.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:03 smithi172 ceph-mon[67937]: pgmap v77: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:05 smithi164 ceph-mon[83622]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:50:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:05 smithi164 ceph-mon[88618]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:50:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:05 smithi172 ceph-mon[67937]: pgmap v78: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:50:08.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:07 smithi164 ceph-mon[83622]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-03-26T02:50:08.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:07 smithi164 ceph-mon[88618]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-03-26T02:50:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:07 smithi172 ceph-mon[67937]: pgmap v79: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 981 B/s rd, 0 op/s 2024-03-26T02:50:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:08 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:09.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:08 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:08 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:09.672 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:50:10.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:09 smithi164 ceph-mon[83622]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:10.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:09 smithi164 ceph-mon[88618]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:10.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:09 smithi172 ceph-mon[67937]: pgmap v80: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:10.326 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:50:10.326 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (4m) 15s ago 26m 18.1M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:50:10.326 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (4m) 2m ago 26m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:50:10.326 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (5m) 15s ago 25m 57.7M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:50:10.326 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (5m) 2m ago 31m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (15m) 15s ago 32m 505M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (2m) 15s ago 32m 45.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (3m) 2m ago 31m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (2m) 15s ago 31m 37.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (5m) 15s ago 27m 8795k - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (5m) 2m ago 27m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (20s) 15s ago 30m 12.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (30m) 15s ago 30m 57.7M 2992M 17.2.0 e1d6a67b021e 59c22704e64a 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (30m) 15s ago 30m 54.1M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (29m) 15s ago 29m 57.4M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (29m) 2m ago 29m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (29m) 2m ago 29m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (28m) 2m ago 28m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:50:10.327 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (28m) 2m ago 28m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:50:10.328 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (5m) 2m ago 27m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:50:10.328 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (26m) 15s ago 26m 78.5M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:50:10.328 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (25m) 2m ago 25m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 1 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:50:10.728 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:50:10.729 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:50:10.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:50:10.729 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:50:10.729 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-03-26T02:50:10.729 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 6 2024-03-26T02:50:10.729 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:50:10.729 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:50:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:10 smithi164 ceph-mon[83622]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:10 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:11.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:10 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/1570884395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:11.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:10 smithi164 ceph-mon[88618]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:11.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:10 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:11.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:10 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/1570884395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:10 smithi172 ceph-mon[67937]: from='client.54282 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:10 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:11.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:10 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/1570884395' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) osd. Upgrade limited to 2 daemons (1 remaining).", 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "1/8 daemons upgraded", 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:50:11.061 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:50:12.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:11 smithi164 ceph-mon[83622]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:12.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:11 smithi164 ceph-mon[83622]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:12.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:11 smithi164 ceph-mon[83622]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:12.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:50:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:50:11] "GET /metrics HTTP/1.1" 200 38113 "" "Prometheus/2.43.0" 2024-03-26T02:50:12.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:11 smithi164 ceph-mon[88618]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:12.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:11 smithi164 ceph-mon[88618]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:12.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:11 smithi164 ceph-mon[88618]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:12.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:11 smithi172 ceph-mon[67937]: from='client.64222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:12.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:11 smithi172 ceph-mon[67937]: from='client.54294 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:12.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:11 smithi172 ceph-mon[67937]: pgmap v81: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:12 smithi164 ceph-mon[83622]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:13.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:12 smithi164 ceph-mon[88618]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:12 smithi172 ceph-mon[67937]: from='client.64240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:13 smithi164 ceph-mon[83622]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:14.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:13 smithi164 ceph-mon[88618]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:13 smithi172 ceph-mon[67937]: pgmap v82: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:16.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:15 smithi164 ceph-mon[83622]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:16.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:16.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:16.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:16.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:15 smithi164 ceph-mon[88618]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:16.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:16.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:16.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:15 smithi172 ceph-mon[67937]: pgmap v83: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[88618]: Upgrade: osd.1 is safe to restart 2024-03-26T02:50:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:50:16.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:16.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:16.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[83622]: Upgrade: osd.1 is safe to restart 2024-03-26T02:50:16.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:16.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:50:16.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:16 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:16 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-03-26T02:50:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:16 smithi172 ceph-mon[67937]: Upgrade: osd.1 is safe to restart 2024-03-26T02:50:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:16 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:16 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T02:50:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:16 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:18.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:17 smithi164 ceph-mon[83622]: Upgrade: Updating osd.1 2024-03-26T02:50:18.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:17 smithi164 ceph-mon[83622]: Deploying daemon osd.1 on smithi164 2024-03-26T02:50:18.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:17 smithi164 ceph-mon[83622]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:17 smithi164 ceph-mon[88618]: Upgrade: Updating osd.1 2024-03-26T02:50:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:17 smithi164 ceph-mon[88618]: Deploying daemon osd.1 on smithi164 2024-03-26T02:50:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:17 smithi164 ceph-mon[88618]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:17 smithi172 ceph-mon[67937]: Upgrade: Updating osd.1 2024-03-26T02:50:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:17 smithi172 ceph-mon[67937]: Deploying daemon osd.1 on smithi164 2024-03-26T02:50:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:17 smithi172 ceph-mon[67937]: pgmap v84: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:18.856 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:18 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:18.856 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:18 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:18 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:19 smithi164 ceph-mon[83622]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:20.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:19 smithi164 ceph-mon[88618]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:19 smithi172 ceph-mon[67937]: pgmap v85: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:20.544 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:20 smithi164 systemd[1]: Stopping Ceph osd.1 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:50:20.544 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[35369]: 2024-03-26T02:50:20.298+0000 7fb7a10e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:50:20.545 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[35369]: 2024-03-26T02:50:20.298+0000 7fb7a10e5700 -1 osd.1 117 *** Got signal Terminated *** 2024-03-26T02:50:20.545 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[35369]: 2024-03-26T02:50:20.298+0000 7fb7a10e5700 -1 osd.1 117 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:50:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:20 smithi164 ceph-mon[83622]: osd.1 marked itself down and dead 2024-03-26T02:50:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:20 smithi164 ceph-mon[88618]: osd.1 marked itself down and dead 2024-03-26T02:50:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:20 smithi172 ceph-mon[67937]: osd.1 marked itself down and dead 2024-03-26T02:50:21.499 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:21 smithi164 podman[100479]: 2024-03-26 02:50:21.177837152 +0000 UTC m=+0.983699805 container died 59c22704e64a94fb465dfe1a73e83a47be4a5aecd64458902a4bbc064eea0d76 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, vcs-type=git, RELEASE=HEAD, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, ceph=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, version=8, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, release=754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-26T02:50:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:21 smithi164 ceph-mon[83622]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:21 smithi164 ceph-mon[83622]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:50:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:21 smithi164 ceph-mon[83622]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T02:50:21.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:21 smithi164 ceph-mon[88618]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:21.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:21 smithi164 ceph-mon[88618]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:50:21.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:21 smithi164 ceph-mon[88618]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T02:50:21.797 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:21 smithi164 podman[100479]: 2024-03-26 02:50:21.606687557 +0000 UTC m=+1.412550219 container cleanup 59c22704e64a94fb465dfe1a73e83a47be4a5aecd64458902a4bbc064eea0d76 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , distribution-scope=public, ceph=True, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, com.redhat.component=centos-stream-container, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-03-26T02:50:21.797 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:21 smithi164 bash[100479]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1 2024-03-26T02:50:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:21 smithi172 ceph-mon[67937]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:21 smithi172 ceph-mon[67937]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:50:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:21 smithi172 ceph-mon[67937]: osdmap e118: 8 total, 7 up, 8 in 2024-03-26T02:50:22.214 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:50:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:50:21] "GET /metrics HTTP/1.1" 200 38111 "" "Prometheus/2.43.0" 2024-03-26T02:50:22.215 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:21 smithi164 podman[100498]: 2024-03-26 02:50:21.79960947 +0000 UTC m=+0.616506617 container remove 59c22704e64a94fb465dfe1a73e83a47be4a5aecd64458902a4bbc064eea0d76 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, ceph=True, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, release=754, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0) 2024-03-26T02:50:22.529 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:22 smithi164 podman[100589]: 2024-03-26 02:50:22.117967746 +0000 UTC m=+0.018107348 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:50:22.529 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:22 smithi164 podman[100589]: 2024-03-26 02:50:22.242251663 +0000 UTC m=+0.142391255 container create 7a2360c07c000d2dc874927f2a75ed69e54771e14bb670425e859325e9f9756b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T02:50:22.529 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:22 smithi164 podman[100589]: 2024-03-26 02:50:22.445466587 +0000 UTC m=+0.345606188 container init 7a2360c07c000d2dc874927f2a75ed69e54771e14bb670425e859325e9f9756b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T02:50:22.529 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:22 smithi164 podman[100589]: 2024-03-26 02:50:22.44898103 +0000 UTC m=+0.349120631 container start 7a2360c07c000d2dc874927f2a75ed69e54771e14bb670425e859325e9f9756b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T02:50:22.529 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:22 smithi164 podman[100589]: 2024-03-26 02:50:22.528449504 +0000 UTC m=+0.428589105 container attach 7a2360c07c000d2dc874927f2a75ed69e54771e14bb670425e859325e9f9756b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T02:50:22.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:22 smithi164 ceph-mon[83622]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T02:50:22.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:22 smithi164 ceph-mon[88618]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T02:50:22.795 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:22 smithi164 podman[100589]: 2024-03-26 02:50:22.743377632 +0000 UTC m=+0.643517231 container died 7a2360c07c000d2dc874927f2a75ed69e54771e14bb670425e859325e9f9756b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T02:50:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:22 smithi172 ceph-mon[67937]: osdmap e119: 8 total, 7 up, 8 in 2024-03-26T02:50:23.758 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:23 smithi164 podman[100615]: 2024-03-26 02:50:23.398896149 +0000 UTC m=+0.649852984 container remove 7a2360c07c000d2dc874927f2a75ed69e54771e14bb670425e859325e9f9756b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T02:50:23.758 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:23 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-26T02:50:23.758 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:23 smithi164 systemd[1]: Stopped Ceph osd.1 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:50:23.758 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:23 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.1.service: Consumed 14.927s CPU time. 2024-03-26T02:50:23.758 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:23 smithi164 systemd[1]: Starting Ceph osd.1 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:50:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:23 smithi164 ceph-mon[83622]: pgmap v89: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:24.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:23 smithi164 ceph-mon[88618]: pgmap v89: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:24.045 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:23 smithi164 podman[100717]: 2024-03-26 02:50:23.74511604 +0000 UTC m=+0.018034985 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:50:24.045 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:23 smithi164 podman[100717]: 2024-03-26 02:50:23.867259566 +0000 UTC m=+0.140178540 container create 707c41a623da12c0db5bb45ef1e45b9df40281077882c90e337cc095c75d5a96 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:50:24.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:23 smithi172 ceph-mon[67937]: pgmap v89: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:24.544 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 podman[100717]: 2024-03-26 02:50:24.105645478 +0000 UTC m=+0.378564406 container init 707c41a623da12c0db5bb45ef1e45b9df40281077882c90e337cc095c75d5a96 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T02:50:24.544 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 podman[100717]: 2024-03-26 02:50:24.109162885 +0000 UTC m=+0.382081823 container start 707c41a623da12c0db5bb45ef1e45b9df40281077882c90e337cc095c75d5a96 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T02:50:24.545 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 podman[100717]: 2024-03-26 02:50:24.21161523 +0000 UTC m=+0.484534159 container attach 707c41a623da12c0db5bb45ef1e45b9df40281077882c90e337cc095c75d5a96 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate[100731]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate[100731]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 bash[100717]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 bash[100717]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 bash[100717]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 bash[100717]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 bash[100717]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 bash[100717]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T02:50:24.812 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 bash[100717]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-26T02:50:24.813 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate[100731]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T02:50:24.813 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate[100731]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T02:50:24.813 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate[100731]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-03-26T02:50:24.813 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate[100731]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-26T02:50:24.813 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate[100731]: --> ceph-volume raw activate successful for osd ID: 1 2024-03-26T02:50:24.813 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:24 smithi164 podman[100717]: 2024-03-26 02:50:24.578039847 +0000 UTC m=+0.850958783 container died 707c41a623da12c0db5bb45ef1e45b9df40281077882c90e337cc095c75d5a96 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T02:50:25.257 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-mon[83622]: pgmap v90: 161 pgs: 1 active+undersized+degraded, 20 stale+active+clean, 5 active+undersized, 135 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:50:25.258 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-mon[83622]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:50:25.258 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-mon[88618]: pgmap v90: 161 pgs: 1 active+undersized+degraded, 20 stale+active+clean, 5 active+undersized, 135 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:50:25.258 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:24 smithi164 ceph-mon[88618]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:50:25.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:24 smithi172 ceph-mon[67937]: pgmap v90: 161 pgs: 1 active+undersized+degraded, 20 stale+active+clean, 5 active+undersized, 135 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:50:25.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:24 smithi172 ceph-mon[67937]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:50:25.544 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:25 smithi164 podman[100862]: 2024-03-26 02:50:25.257375053 +0000 UTC m=+0.674065896 container remove 707c41a623da12c0db5bb45ef1e45b9df40281077882c90e337cc095c75d5a96 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T02:50:25.544 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:25 smithi164 podman[100927]: 2024-03-26 02:50:25.466647181 +0000 UTC m=+0.112903595 container create 7fb9fd299fb06725571330d0a774a87cb0f87596d9773789971c32ea9cf09ce3 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:50:25.544 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:25 smithi164 podman[100927]: 2024-03-26 02:50:25.372144533 +0000 UTC m=+0.018400918 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:50:26.044 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:25 smithi164 podman[100927]: 2024-03-26 02:50:25.702292702 +0000 UTC m=+0.348549074 container init 7fb9fd299fb06725571330d0a774a87cb0f87596d9773789971c32ea9cf09ce3 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T02:50:26.044 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:25 smithi164 podman[100927]: 2024-03-26 02:50:25.705333772 +0000 UTC m=+0.351590145 container start 7fb9fd299fb06725571330d0a774a87cb0f87596d9773789971c32ea9cf09ce3 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T02:50:26.045 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:25 smithi164 ceph-osd[100946]: -- 172.21.15.164:0/2671818745 <== mon.1 v2:172.21.15.164:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bff314f680 con 0x55bff3118c00 2024-03-26T02:50:26.045 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:25 smithi164 bash[100927]: 7fb9fd299fb06725571330d0a774a87cb0f87596d9773789971c32ea9cf09ce3 2024-03-26T02:50:26.045 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:25 smithi164 systemd[1]: Started Ceph osd.1 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:50:26.544 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[100942]: 2024-03-26T02:50:26.256+0000 7f9c980f47c0 -1 Falling back to public interface 2024-03-26T02:50:27.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:27.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[83622]: pgmap v91: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:26 smithi164 ceph-mon[88618]: pgmap v91: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:27.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:26 smithi172 ceph-mon[67937]: pgmap v91: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:28.237 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:27 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:28.237 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:27 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:28.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:27 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:29.217 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:28 smithi164 ceph-mon[83622]: pgmap v92: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:29.217 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:28 smithi164 ceph-mon[88618]: pgmap v92: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:29.217 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:28 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[100942]: 2024-03-26T02:50:28.964+0000 7f9c980f47c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-03-26T02:50:29.217 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:28 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[100942]: 2024-03-26T02:50:28.999+0000 7f9c980f47c0 -1 osd.1 117 log_to_monitors true 2024-03-26T02:50:29.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:28 smithi172 ceph-mon[67937]: pgmap v92: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 121 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:30.044 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:50:29 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[100942]: 2024-03-26T02:50:29.930+0000 7f9c8954c700 -1 osd.1 117 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:50:30.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:29 smithi164 ceph-mon[83622]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:50:30.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:29 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:30.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:29 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:29 smithi164 ceph-mon[88618]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:50:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:29 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:29 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:30.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:29 smithi172 ceph-mon[67937]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T02:50:30.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:29 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:30.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:29 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[83622]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:50:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[83622]: osdmap e120: 8 total, 7 up, 8 in 2024-03-26T02:50:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[83622]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[83622]: pgmap v94: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[88618]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[88618]: osdmap e120: 8 total, 7 up, 8 in 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[88618]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:30 smithi164 ceph-mon[88618]: pgmap v94: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:30 smithi172 ceph-mon[67937]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T02:50:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:30 smithi172 ceph-mon[67937]: osdmap e120: 8 total, 7 up, 8 in 2024-03-26T02:50:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:30 smithi172 ceph-mon[67937]: from='osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:50:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:30 smithi172 ceph-mon[67937]: pgmap v94: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 99/723 objects degraded (13.693%) 2024-03-26T02:50:32.231 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:50:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:50:31] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.43.0" 2024-03-26T02:50:32.231 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[83622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:50:32.231 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[83622]: osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909] boot 2024-03-26T02:50:32.231 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[83622]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[83622]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[88618]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[88618]: osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909] boot 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[88618]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:32.232 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:31 smithi164 ceph-mon[88618]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T02:50:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:31 smithi172 ceph-mon[67937]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:50:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:31 smithi172 ceph-mon[67937]: osd.1 [v2:172.21.15.164:6810/849626909,v1:172.21.15.164:6811/849626909] boot 2024-03-26T02:50:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:31 smithi172 ceph-mon[67937]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T02:50:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:50:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:32.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:31 smithi172 ceph-mon[67937]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T02:50:33.186 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:32 smithi164 ceph-mon[83622]: pgmap v97: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 99/723 objects degraded (13.693%) 2024-03-26T02:50:33.186 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:32 smithi164 ceph-mon[88618]: pgmap v97: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 99/723 objects degraded (13.693%) 2024-03-26T02:50:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:32 smithi172 ceph-mon[67937]: pgmap v97: 161 pgs: 19 active+undersized+degraded, 44 active+undersized, 98 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 99/723 objects degraded (13.693%) 2024-03-26T02:50:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:34 smithi164 ceph-mon[83622]: Health check update: Degraded data redundancy: 99/723 objects degraded (13.693%), 19 pgs degraded (PG_DEGRADED) 2024-03-26T02:50:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:34 smithi164 ceph-mon[88618]: Health check update: Degraded data redundancy: 99/723 objects degraded (13.693%), 19 pgs degraded (PG_DEGRADED) 2024-03-26T02:50:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:34 smithi172 ceph-mon[67937]: Health check update: Degraded data redundancy: 99/723 objects degraded (13.693%), 19 pgs degraded (PG_DEGRADED) 2024-03-26T02:50:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: pgmap v98: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 93/723 objects degraded (12.863%) 2024-03-26T02:50:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:50:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: pgmap v98: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 93/723 objects degraded (12.863%) 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:50:36.048 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:50:36.050 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:50:36.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.053 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: pgmap v98: 161 pgs: 18 active+undersized+degraded, 40 active+undersized, 103 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 93/723 objects degraded (12.863%) 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:50:36.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:50:36.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:37.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:36 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:50:37.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:36 smithi164 ceph-mon[83622]: Upgrade: Finalizing container_image settings 2024-03-26T02:50:37.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:36 smithi164 ceph-mon[83622]: Upgrade: Complete! 2024-03-26T02:50:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:36 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:50:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:36 smithi164 ceph-mon[88618]: Upgrade: Finalizing container_image settings 2024-03-26T02:50:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:36 smithi164 ceph-mon[88618]: Upgrade: Complete! 2024-03-26T02:50:37.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:36 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:50:37.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:36 smithi172 ceph-mon[67937]: Upgrade: Finalizing container_image settings 2024-03-26T02:50:37.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:36 smithi172 ceph-mon[67937]: Upgrade: Complete! 2024-03-26T02:50:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:37 smithi164 ceph-mon[83622]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:50:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:37 smithi164 ceph-mon[83622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 93/723 objects degraded (12.863%), 18 pgs degraded) 2024-03-26T02:50:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:37 smithi164 ceph-mon[83622]: Cluster is now healthy 2024-03-26T02:50:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:37 smithi164 ceph-mon[88618]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:50:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:37 smithi164 ceph-mon[88618]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 93/723 objects degraded (12.863%), 18 pgs degraded) 2024-03-26T02:50:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:37 smithi164 ceph-mon[88618]: Cluster is now healthy 2024-03-26T02:50:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:37 smithi172 ceph-mon[67937]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:50:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:37 smithi172 ceph-mon[67937]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 93/723 objects degraded (12.863%), 18 pgs degraded) 2024-03-26T02:50:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:37 smithi172 ceph-mon[67937]: Cluster is now healthy 2024-03-26T02:50:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:38 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:38 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:38 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:39 smithi164 ceph-mon[83622]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:39 smithi164 ceph-mon[88618]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:39 smithi172 ceph-mon[67937]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:42.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:41 smithi164 ceph-mon[83622]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:50:42.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:41 smithi164 ceph-mon[88618]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:50:42.024 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:50:42.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:41 smithi172 ceph-mon[67937]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:50:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:50:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:50:42] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.43.0" 2024-03-26T02:50:43.023 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:42 smithi164 ceph-mon[88618]: from='client.54318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:43.024 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:42 smithi164 ceph-mon[83622]: from='client.54318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:42 smithi172 ceph-mon[67937]: from='client.54318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:44.023 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:43 smithi164 ceph-mon[83622]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-26T02:50:44.024 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:43 smithi164 ceph-mon[88618]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-26T02:50:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:43 smithi172 ceph-mon[67937]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 949 B/s rd, 0 op/s 2024-03-26T02:50:44.081 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (5m) 13s ago 27m 18.2M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (5m) 2m ago 27m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (6m) 13s ago 26m 58.5M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (6m) 2m ago 31m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (16m) 13s ago 33m 507M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (3m) 13s ago 33m 46.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (3m) 2m ago 32m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (2m) 13s ago 32m 39.3M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (6m) 13s ago 28m 16.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (5m) 2m ago 27m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:50:44.082 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (54s) 13s ago 31m 50.9M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (18s) 13s ago 31m 12.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (30m) 13s ago 30m 55.2M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (30m) 13s ago 30m 58.9M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (29m) 2m ago 29m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (29m) 2m ago 29m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (29m) 2m ago 29m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (28m) 2m ago 28m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (5m) 2m ago 28m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (26m) 13s ago 26m 78.7M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:50:44.083 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (26m) 2m ago 26m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:50:44.756 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-26T02:50:46.031 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[83622]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:46.031 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[83622]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:46.031 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:46.031 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:46.031 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:46.031 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[88618]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:46.032 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[88618]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:46.032 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:46.032 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:46.032 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:45 smithi172 ceph-mon[67937]: from='client.54324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:45 smithi172 ceph-mon[67937]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:50:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:50:46.853 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:50:47.445 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 7'"'"'' 2024-03-26T02:50:47.992 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:47 smithi164 ceph-mon[83622]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:47.992 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:47 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2577041230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:47.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:47 smithi164 ceph-mon[88618]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:47.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:47 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2577041230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:48.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:47 smithi172 ceph-mon[67937]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:48.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:47 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2577041230' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:50:48.982 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:48 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:48.982 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:48 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:48 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:49 smithi164 ceph-mon[83622]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:49 smithi164 ceph-mon[88618]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:49 smithi172 ceph-mon[67937]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:50 smithi164 ceph-mon[83622]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:50 smithi164 ceph-mon[88618]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:50 smithi172 ceph-mon[67937]: from='client.64270 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:51.567 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:50:52.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:51 smithi164 ceph-mon[88618]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:52.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:51 smithi164 ceph-mon[83622]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:52.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:51 smithi172 ceph-mon[67937]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:50:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:50:52] "GET /metrics HTTP/1.1" 200 38184 "" "Prometheus/2.43.0" 2024-03-26T02:50:52.368 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade status' 2024-03-26T02:50:53.904 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:53 smithi164 ceph-mon[83622]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:53.904 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:53 smithi164 ceph-mon[88618]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:54.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:53 smithi172 ceph-mon[67937]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:54.376 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:50:54.376 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": null, 2024-03-26T02:50:54.376 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": false, 2024-03-26T02:50:54.376 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "", 2024-03-26T02:50:54.376 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:50:54.377 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": null, 2024-03-26T02:50:54.377 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:50:54.377 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:50:54.377 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:50:55.053 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:50:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:55 smithi164 ceph-mon[83622]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:55 smithi164 ceph-mon[83622]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:55 smithi164 ceph-mon[88618]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:56.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:55 smithi164 ceph-mon[88618]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:55 smithi172 ceph-mon[67937]: from='client.64276 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:50:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:55 smithi172 ceph-mon[67937]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:50:57.170 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:50:57.801 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd --limit 1' 2024-03-26T02:50:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:57 smithi164 ceph-mon[83622]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:57 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2153821547' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:50:58.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:57 smithi164 ceph-mon[88618]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:58.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:57 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2153821547' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:50:58.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:57 smithi172 ceph-mon[67937]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:50:58.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:57 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2153821547' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:50:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:58 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:59.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:58 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:50:59.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:58 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:00.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:59 smithi164 ceph-mon[83622]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:00.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:50:59 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:00.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:59 smithi164 ceph-mon[88618]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:00.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:50:59 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:00.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:59 smithi172 ceph-mon[67937]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:00.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:50:59 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:02.014 INFO:teuthology.orchestra.run.smithi164.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:02.015 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:51:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:51:01] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-03-26T02:51:02.015 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:01 smithi164 ceph-mon[83622]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:02.015 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:01 smithi164 ceph-mon[83622]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:02.015 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:01 smithi164 ceph-mon[88618]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:02.015 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:01 smithi164 ceph-mon[88618]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:02.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:01 smithi172 ceph-mon[67937]: from='client.54354 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "crash,osd", "limit": 1, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:02.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:01 smithi172 ceph-mon[67937]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:02.693 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-26T02:51:03.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[83622]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[83622]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[83622]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[88618]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:03.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:03.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:03.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:03.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:03.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[88618]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:03.296 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:03 smithi164 ceph-mon[88618]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:03 smithi172 ceph-mon[67937]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:03 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:03 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:03 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:03 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:03 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:03 smithi172 ceph-mon[67937]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:03 smithi172 ceph-mon[67937]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:04.664 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:51:05.331 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:51:05.331 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (5m) 35s ago 27m 18.2M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:51:05.331 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (5m) 3m ago 27m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:51:05.331 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (6m) 35s ago 26m 58.5M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:51:05.331 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (6m) 3m ago 32m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (16m) 35s ago 33m 507M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (3m) 35s ago 33m 46.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (4m) 3m ago 32m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (3m) 35s ago 32m 39.3M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (6m) 35s ago 28m 16.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (6m) 3m ago 28m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (75s) 35s ago 31m 50.9M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (39s) 35s ago 31m 12.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (31m) 35s ago 31m 55.2M 2992M 17.2.0 e1d6a67b021e 630f5ded884b 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (30m) 35s ago 30m 58.9M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (30m) 3m ago 30m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (29m) 3m ago 29m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (29m) 3m ago 29m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (29m) 3m ago 29m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (6m) 3m ago 28m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (26m) 35s ago 26m 78.7M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:51:05.332 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (26m) 3m ago 26m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:51:05.755 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:51:05.755 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:51:05.756 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:51:05.757 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-03-26T02:51:05.757 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 7 2024-03-26T02:51:05.757 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:51:05.757 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:51:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:05 smithi164 ceph-mon[83622]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:06.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:05 smithi164 ceph-mon[83622]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:05 smithi164 ceph-mon[88618]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:05 smithi164 ceph-mon[88618]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:06.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:05 smithi172 ceph-mon[67937]: from='client.54360 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:06.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:05 smithi172 ceph-mon[67937]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:06.098 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:51:06.098 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", 2024-03-26T02:51:06.098 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:51:06.098 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) crash,osd. Upgrade limited to 1 daemons (1 remaining).", 2024-03-26T02:51:06.098 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:51:06.099 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "", 2024-03-26T02:51:06.099 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 image", 2024-03-26T02:51:06.099 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:51:06.099 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:51:06.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:06.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:06.751 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2704627867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2704627867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:07.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:51:07.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='client.64300 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='client.54372 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2704627867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:07.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:51:08.039 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:08.039 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:51:08.039 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mgr 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mon 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all crash 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: Upgrade: osd.2 is safe to restart 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[83622]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mgr 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mon 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all crash 2024-03-26T02:51:08.040 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:51:08.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: Upgrade: osd.2 is safe to restart 2024-03-26T02:51:08.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:07 smithi164 ceph-mon[88618]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: from='client.54384 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:51:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:51:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mgr 2024-03-26T02:51:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mon 2024-03-26T02:51:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all crash 2024-03-26T02:51:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-03-26T02:51:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: Upgrade: osd.2 is safe to restart 2024-03-26T02:51:08.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:07 smithi172 ceph-mon[67937]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:08.976 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:08 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:08.976 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:08 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:08.976 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:08 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:51:08.976 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:08 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:08.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:08 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:08.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:08 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:08.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:08 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:51:08.976 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:08 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:08 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:08 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:08 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T02:51:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:08 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:09.934 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:09 smithi164 ceph-mon[83622]: Upgrade: Updating osd.2 2024-03-26T02:51:09.934 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:09 smithi164 ceph-mon[83622]: Deploying daemon osd.2 on smithi164 2024-03-26T02:51:09.934 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:09 smithi164 ceph-mon[83622]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:09.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:09 smithi164 ceph-mon[88618]: Upgrade: Updating osd.2 2024-03-26T02:51:09.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:09 smithi164 ceph-mon[88618]: Deploying daemon osd.2 on smithi164 2024-03-26T02:51:09.934 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:09 smithi164 ceph-mon[88618]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:09 smithi172 ceph-mon[67937]: Upgrade: Updating osd.2 2024-03-26T02:51:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:09 smithi172 ceph-mon[67937]: Deploying daemon osd.2 on smithi164 2024-03-26T02:51:10.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:09 smithi172 ceph-mon[67937]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:11.594 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:11 smithi164 systemd[1]: Stopping Ceph osd.2 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:51:12.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:11 smithi164 ceph-mon[83622]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:12.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:11 smithi164 ceph-mon[83622]: osd.2 marked itself down and dead 2024-03-26T02:51:12.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:11 smithi164 ceph-mon[88618]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:12.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:11 smithi164 ceph-mon[88618]: osd.2 marked itself down and dead 2024-03-26T02:51:12.001 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[38296]: 2024-03-26T02:51:11.591+0000 7f811ba2c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:51:12.001 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[38296]: 2024-03-26T02:51:11.591+0000 7f811ba2c700 -1 osd.2 122 *** Got signal Terminated *** 2024-03-26T02:51:12.001 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[38296]: 2024-03-26T02:51:11.591+0000 7f811ba2c700 -1 osd.2 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:51:12.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:11 smithi172 ceph-mon[67937]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:12.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:11 smithi172 ceph-mon[67937]: osd.2 marked itself down and dead 2024-03-26T02:51:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:51:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:51:11] "GET /metrics HTTP/1.1" 200 38180 "" "Prometheus/2.43.0" 2024-03-26T02:51:12.294 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:12 smithi164 podman[106665]: 2024-03-26 02:51:12.245322868 +0000 UTC m=+0.783914435 container died 630f5ded884bf380ac1f6ee8ff47621c1a2e62fe1f58ae35b420152e18ec31c1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, distribution-scope=public, GIT_BRANCH=HEAD, ceph=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-type=git, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-26T02:51:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:12 smithi164 ceph-mon[83622]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:51:12.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:12 smithi164 ceph-mon[83622]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T02:51:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:12 smithi164 ceph-mon[88618]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:51:12.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:12 smithi164 ceph-mon[88618]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T02:51:12.795 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:12 smithi164 podman[106665]: 2024-03-26 02:51:12.669956398 +0000 UTC m=+1.208548015 container cleanup 630f5ded884bf380ac1f6ee8ff47621c1a2e62fe1f58ae35b420152e18ec31c1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, release=754, version=8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vcs-type=git, ceph=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux ) 2024-03-26T02:51:12.795 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:12 smithi164 bash[106665]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2 2024-03-26T02:51:13.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:12 smithi172 ceph-mon[67937]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:51:13.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:12 smithi172 ceph-mon[67937]: osdmap e123: 8 total, 7 up, 8 in 2024-03-26T02:51:13.294 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:12 smithi164 podman[106684]: 2024-03-26 02:51:12.896759166 +0000 UTC m=+0.650109648 container remove 630f5ded884bf380ac1f6ee8ff47621c1a2e62fe1f58ae35b420152e18ec31c1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.openshift.expose-services=, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_BRANCH=HEAD, ceph=True, com.redhat.component=centos-stream-container, GIT_CLEAN=True, vendor=Red Hat, Inc.) 2024-03-26T02:51:13.650 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:13 smithi164 podman[106777]: 2024-03-26 02:51:13.331095353 +0000 UTC m=+0.111010347 container create 1634be124549b494b17567772cbe17db9a191f554eb985da43949b100e0ec258 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:51:13.650 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:13 smithi164 podman[106777]: 2024-03-26 02:51:13.238258639 +0000 UTC m=+0.018173633 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:51:13.651 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:13 smithi164 podman[106777]: 2024-03-26 02:51:13.566774334 +0000 UTC m=+0.346689329 container init 1634be124549b494b17567772cbe17db9a191f554eb985da43949b100e0ec258 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T02:51:13.651 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:13 smithi164 podman[106777]: 2024-03-26 02:51:13.571280895 +0000 UTC m=+0.351195892 container start 1634be124549b494b17567772cbe17db9a191f554eb985da43949b100e0ec258 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:51:13.651 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:13 smithi164 podman[106777]: 2024-03-26 02:51:13.65007186 +0000 UTC m=+0.429986879 container attach 1634be124549b494b17567772cbe17db9a191f554eb985da43949b100e0ec258 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T02:51:14.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:13 smithi164 ceph-mon[83622]: pgmap v118: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:51:14.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:13 smithi164 ceph-mon[83622]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T02:51:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:13 smithi164 ceph-mon[88618]: pgmap v118: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:51:14.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:13 smithi164 ceph-mon[88618]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T02:51:14.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:13 smithi164 podman[106777]: 2024-03-26 02:51:13.884685619 +0000 UTC m=+0.664600619 container died 1634be124549b494b17567772cbe17db9a191f554eb985da43949b100e0ec258 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T02:51:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:13 smithi172 ceph-mon[67937]: pgmap v118: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:51:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:13 smithi172 ceph-mon[67937]: osdmap e124: 8 total, 7 up, 8 in 2024-03-26T02:51:15.018 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:14 smithi164 podman[106813]: 2024-03-26 02:51:14.570981874 +0000 UTC m=+0.681642485 container remove 1634be124549b494b17567772cbe17db9a191f554eb985da43949b100e0ec258 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T02:51:15.018 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:14 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-26T02:51:15.018 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:14 smithi164 systemd[1]: Stopped Ceph osd.2 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:51:15.018 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:14 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.2.service: Consumed 12.362s CPU time. 2024-03-26T02:51:15.018 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:14 smithi164 systemd[1]: Starting Ceph osd.2 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:51:15.284 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 podman[106919]: 2024-03-26 02:51:14.920188489 +0000 UTC m=+0.017865675 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:51:15.284 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 podman[106919]: 2024-03-26 02:51:15.06470628 +0000 UTC m=+0.162383450 container create d84d7584d647d2d2f37ac9a84a4f36c7e85aef06253f957a3e4bea0c36b0433e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:51:15.284 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 podman[106919]: 2024-03-26 02:51:15.283474742 +0000 UTC m=+0.381151910 container init d84d7584d647d2d2f37ac9a84a4f36c7e85aef06253f957a3e4bea0c36b0433e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T02:51:15.544 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 podman[106919]: 2024-03-26 02:51:15.286759262 +0000 UTC m=+0.384436440 container start d84d7584d647d2d2f37ac9a84a4f36c7e85aef06253f957a3e4bea0c36b0433e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T02:51:15.544 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 podman[106919]: 2024-03-26 02:51:15.341741888 +0000 UTC m=+0.439419065 container attach d84d7584d647d2d2f37ac9a84a4f36c7e85aef06253f957a3e4bea0c36b0433e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:51:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-mon[83622]: pgmap v120: 161 pgs: 1 active+undersized+degraded, 10 stale+active+clean, 7 active+undersized, 143 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:51:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-mon[83622]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:51:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-mon[88618]: pgmap v120: 161 pgs: 1 active+undersized+degraded, 10 stale+active+clean, 7 active+undersized, 143 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:51:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-mon[88618]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:51:16.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate[106934]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T02:51:16.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate[106934]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T02:51:16.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 bash[106919]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T02:51:16.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 bash[106919]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 bash[106919]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 bash[106919]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 bash[106919]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 bash[106919]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 bash[106919]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate[106934]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate[106934]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate[106934]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate[106934]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate[106934]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T02:51:16.046 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:15 smithi164 podman[106919]: 2024-03-26 02:51:15.752846996 +0000 UTC m=+0.850524168 container died d84d7584d647d2d2f37ac9a84a4f36c7e85aef06253f957a3e4bea0c36b0433e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T02:51:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:15 smithi172 ceph-mon[67937]: pgmap v120: 161 pgs: 1 active+undersized+degraded, 10 stale+active+clean, 7 active+undersized, 143 active+clean; 585 KiB data, 141 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:51:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:15 smithi172 ceph-mon[67937]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:51:16.693 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:16 smithi164 podman[107069]: 2024-03-26 02:51:16.371482711 +0000 UTC m=+0.613192704 container remove d84d7584d647d2d2f37ac9a84a4f36c7e85aef06253f957a3e4bea0c36b0433e (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T02:51:16.693 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:16 smithi164 podman[107133]: 2024-03-26 02:51:16.580633491 +0000 UTC m=+0.113412031 container create f4dde90188ad9316bc905687072e917b96fbb74b39e767fc14eeb61f65f70454 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T02:51:16.693 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:16 smithi164 podman[107133]: 2024-03-26 02:51:16.48480824 +0000 UTC m=+0.017586778 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:51:17.044 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:16 smithi164 podman[107133]: 2024-03-26 02:51:16.791692941 +0000 UTC m=+0.324471479 container init f4dde90188ad9316bc905687072e917b96fbb74b39e767fc14eeb61f65f70454 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:51:17.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:16 smithi164 podman[107133]: 2024-03-26 02:51:16.794763227 +0000 UTC m=+0.327541765 container start f4dde90188ad9316bc905687072e917b96fbb74b39e767fc14eeb61f65f70454 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T02:51:17.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:16 smithi164 ceph-osd[107152]: -- 172.21.15.164:0/3928836688 <== mon.0 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56194fde7680 con 0x56194fdb1400 2024-03-26T02:51:17.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:16 smithi164 bash[107133]: f4dde90188ad9316bc905687072e917b96fbb74b39e767fc14eeb61f65f70454 2024-03-26T02:51:17.045 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:16 smithi164 systemd[1]: Started Ceph osd.2 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:51:17.544 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[107148]: 2024-03-26T02:51:17.333+0000 7f5338d747c0 -1 Falling back to public interface 2024-03-26T02:51:17.970 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[83622]: pgmap v121: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:17.970 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:17.970 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:17.970 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:17.970 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:17.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[88618]: pgmap v121: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:17.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:17.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:17.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:17.971 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:17 smithi172 ceph-mon[67937]: pgmap v121: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:18 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:18 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:18 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:20.039 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:19 smithi164 ceph-mon[83622]: pgmap v122: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:20.039 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:19 smithi164 ceph-mon[88618]: pgmap v122: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:19 smithi172 ceph-mon[67937]: pgmap v122: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:20.294 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[107148]: 2024-03-26T02:51:20.038+0000 7f5338d747c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-03-26T02:51:20.294 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:20 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[107148]: 2024-03-26T02:51:20.064+0000 7f5338d747c0 -1 osd.2 122 log_to_monitors true 2024-03-26T02:51:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:20 smithi164 ceph-mon[83622]: from='osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:51:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:20 smithi164 ceph-mon[83622]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:51:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:20 smithi164 ceph-mon[88618]: from='osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:51:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:20 smithi164 ceph-mon[88618]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:51:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:20 smithi172 ceph-mon[67937]: from='osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:51:21.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:20 smithi172 ceph-mon[67937]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T02:51:21.544 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[107148]: 2024-03-26T02:51:21.216+0000 7f53261c4700 -1 osd.2 122 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:51:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[88618]: pgmap v123: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[88618]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:51:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[88618]: from='osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:51:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[88618]: osdmap e125: 8 total, 7 up, 8 in 2024-03-26T02:51:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[88618]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:51:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[88618]: from='osd.2 ' entity='osd.2' 2024-03-26T02:51:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[83622]: pgmap v123: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[83622]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[83622]: from='osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[83622]: osdmap e125: 8 total, 7 up, 8 in 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[83622]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[83622]: from='osd.2 ' entity='osd.2' 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:21 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:21 smithi172 ceph-mon[67937]: pgmap v123: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 142 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:21 smithi172 ceph-mon[67937]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T02:51:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:21 smithi172 ceph-mon[67937]: from='osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:51:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:21 smithi172 ceph-mon[67937]: osdmap e125: 8 total, 7 up, 8 in 2024-03-26T02:51:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:21 smithi172 ceph-mon[67937]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:51:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:21 smithi172 ceph-mon[67937]: from='osd.2 ' entity='osd.2' 2024-03-26T02:51:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:21 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:21 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:51:22] "GET /metrics HTTP/1.1" 200 38187 "" "Prometheus/2.43.0" 2024-03-26T02:51:22.987 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[83622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:51:22.987 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[83622]: osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144] boot 2024-03-26T02:51:22.987 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[83622]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[88618]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[88618]: osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144] boot 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[88618]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:22.988 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:23.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:22 smithi172 ceph-mon[67937]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:51:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:22 smithi172 ceph-mon[67937]: osd.2 [v2:172.21.15.164:6818/4226622144,v1:172.21.15.164:6819/4226622144] boot 2024-03-26T02:51:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:22 smithi172 ceph-mon[67937]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T02:51:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:51:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:24.002 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:23 smithi164 ceph-mon[83622]: pgmap v126: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:24.002 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:23 smithi164 ceph-mon[83622]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T02:51:24.002 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:23 smithi164 ceph-mon[88618]: pgmap v126: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:24.002 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:23 smithi164 ceph-mon[88618]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T02:51:24.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:23 smithi172 ceph-mon[67937]: pgmap v126: 161 pgs: 13 active+undersized+degraded, 33 active+undersized, 115 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 65/723 objects degraded (8.990%) 2024-03-26T02:51:24.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:23 smithi172 ceph-mon[67937]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T02:51:24.964 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:24 smithi164 ceph-mon[83622]: Health check update: Degraded data redundancy: 65/723 objects degraded (8.990%), 13 pgs degraded (PG_DEGRADED) 2024-03-26T02:51:24.965 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:24 smithi164 ceph-mon[88618]: Health check update: Degraded data redundancy: 65/723 objects degraded (8.990%), 13 pgs degraded (PG_DEGRADED) 2024-03-26T02:51:25.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:24 smithi172 ceph-mon[67937]: Health check update: Degraded data redundancy: 65/723 objects degraded (8.990%), 13 pgs degraded (PG_DEGRADED) 2024-03-26T02:51:25.887 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:25 smithi164 ceph-mon[83622]: pgmap v128: 161 pgs: 12 active+undersized+degraded, 28 active+undersized, 121 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/723 objects degraded (8.160%) 2024-03-26T02:51:25.887 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:25 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:25.887 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:25 smithi164 ceph-mon[88618]: pgmap v128: 161 pgs: 12 active+undersized+degraded, 28 active+undersized, 121 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/723 objects degraded (8.160%) 2024-03-26T02:51:25.887 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:25 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:25 smithi172 ceph-mon[67937]: pgmap v128: 161 pgs: 12 active+undersized+degraded, 28 active+undersized, 121 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 59/723 objects degraded (8.160%) 2024-03-26T02:51:26.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:25 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: Upgrade: Finalizing container_image settings 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: Upgrade: Complete! 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.798 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: Upgrade: Finalizing container_image settings 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:51:27.800 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: Upgrade: Complete! 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:51:27.801 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:27.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.802 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: Upgrade: Finalizing container_image settings 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:51:27.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:51:27.809 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: Upgrade: Complete! 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:27.810 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:28.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:28 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:28.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:28 smithi164 ceph-mon[83622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/723 objects degraded (8.160%), 12 pgs degraded) 2024-03-26T02:51:28.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:28 smithi164 ceph-mon[83622]: Cluster is now healthy 2024-03-26T02:51:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:28 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:28 smithi164 ceph-mon[88618]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/723 objects degraded (8.160%), 12 pgs degraded) 2024-03-26T02:51:28.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:28 smithi164 ceph-mon[88618]: Cluster is now healthy 2024-03-26T02:51:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:28 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:28 smithi172 ceph-mon[67937]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 59/723 objects degraded (8.160%), 12 pgs degraded) 2024-03-26T02:51:28.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:28 smithi172 ceph-mon[67937]: Cluster is now healthy 2024-03-26T02:51:29.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:29 smithi164 ceph-mon[83622]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-03-26T02:51:29.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:29 smithi164 ceph-mon[88618]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-03-26T02:51:29.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:29 smithi172 ceph-mon[67937]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-03-26T02:51:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:31 smithi164 ceph-mon[83622]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:31 smithi164 ceph-mon[88618]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:31 smithi172 ceph-mon[67937]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:51:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:51:32] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-03-26T02:51:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:33 smithi164 ceph-mon[83622]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:51:34.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:33 smithi164 ceph-mon[88618]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:51:34.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:33 smithi172 ceph-mon[67937]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:51:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:35 smithi164 ceph-mon[83622]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-03-26T02:51:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:35 smithi164 ceph-mon[88618]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-03-26T02:51:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:35 smithi172 ceph-mon[67937]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 891 B/s rd, 0 op/s 2024-03-26T02:51:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:37.255 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:51:37.815 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:37 smithi164 ceph-mon[83622]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:37.815 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:37 smithi164 ceph-mon[83622]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:37.815 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:37 smithi164 ceph-mon[88618]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:37.815 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:37 smithi164 ceph-mon[88618]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:37 smithi172 ceph-mon[67937]: from='client.64330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:37 smithi172 ceph-mon[67937]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:38.962 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:38 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:38.962 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:38 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:38 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:39.394 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (6m) 18s ago 28m 26.9M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (6m) 3m ago 27m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (7m) 18s ago 27m 58.5M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (7m) 3m ago 32m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (17m) 18s ago 34m 509M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (4m) 18s ago 34m 49.7M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (4m) 3m ago 32m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (3m) 18s ago 33m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (6m) 18s ago 28m 16.2M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (6m) 3m ago 28m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (109s) 18s ago 32m 52.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (74s) 18s ago 32m 44.6M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:51:39.395 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (22s) 18s ago 31m 17.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:51:39.396 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (31m) 18s ago 31m 59.6M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:51:39.396 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (30m) 3m ago 30m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:51:39.396 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (30m) 3m ago 30m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:51:39.396 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (30m) 3m ago 30m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:51:39.396 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (29m) 3m ago 29m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:51:39.396 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (6m) 3m ago 29m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:51:39.396 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (27m) 18s ago 27m 78.9M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:51:39.396 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (27m) 3m ago 27m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:51:40.031 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:39 smithi164 ceph-mon[83622]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:40.031 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:39 smithi164 ceph-mon[88618]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:40.054 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 2'"'"'' 2024-03-26T02:51:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:39 smithi172 ceph-mon[67937]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:40 smithi164 ceph-mon[83622]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:41.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:40 smithi164 ceph-mon[88618]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:40 smithi172 ceph-mon[67937]: from='client.54402 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:42.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:41 smithi164 ceph-mon[83622]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:42.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:41 smithi164 ceph-mon[88618]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:42.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:41 smithi172 ceph-mon[67937]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:42.168 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:51:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:51:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:51:41] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-03-26T02:51:42.829 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '"'"'.up_to_date | length == 8'"'"'' 2024-03-26T02:51:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:42 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/154573410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:42 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/154573410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:42 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/154573410' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:51:44.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:43 smithi164 ceph-mon[88618]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:43 smithi164 ceph-mon[83622]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:43 smithi172 ceph-mon[67937]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:45.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:44 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:45.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:45.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:51:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:45 smithi164 ceph-mon[83622]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:45 smithi164 ceph-mon[88618]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:45 smithi172 ceph-mon[67937]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:47.042 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:46 smithi164 ceph-mon[88618]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:47.042 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:46 smithi164 ceph-mon[83622]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:46 smithi172 ceph-mon[67937]: from='client.54414 -' entity='client.admin' cmd=[{"prefix": "orch upgrade check", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:47.103 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:51:47.705 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade status' 2024-03-26T02:51:48.016 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:47 smithi164 ceph-mon[83622]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:48.016 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:47 smithi164 ceph-mon[88618]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:48.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:47 smithi172 ceph-mon[67937]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:49.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:48 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:48 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:48 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:49.732 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:51:49.733 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": null, 2024-03-26T02:51:49.733 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": false, 2024-03-26T02:51:49.733 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "", 2024-03-26T02:51:49.733 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:51:49.733 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": null, 2024-03-26T02:51:49.733 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:51:49.733 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:51:49.733 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:51:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:49 smithi164 ceph-mon[83622]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:50.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:49 smithi164 ceph-mon[88618]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:49 smithi172 ceph-mon[67937]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:50.361 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:51:50.922 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:50 smithi164 ceph-mon[83622]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:50.923 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:50 smithi164 ceph-mon[88618]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:50 smithi172 ceph-mon[67937]: from='client.54420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:51.988 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:51 smithi164 ceph-mon[83622]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:51.988 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:51 smithi164 ceph-mon[88618]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:52.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:51 smithi172 ceph-mon[67937]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:51:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:51:51] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.43.0" 2024-03-26T02:51:52.495 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:51:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:52 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/3648561070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:51:53.176 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:52 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/3648561070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:51:53.176 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:52 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/3648561070' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:51:53.200 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types crash,osd' 2024-03-26T02:51:54.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:53 smithi164 ceph-mon[83622]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:53 smithi164 ceph-mon[88618]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:54.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:53 smithi172 ceph-mon[67937]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:55.030 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:54 smithi164 ceph-mon[83622]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:55.030 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:54 smithi164 ceph-mon[88618]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:54 smithi172 ceph-mon[67937]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:56.229 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:55 smithi164 ceph-mon[83622]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:56.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:55 smithi164 ceph-mon[88618]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:56.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:55 smithi172 ceph-mon[67937]: from='client.64366 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "daemon_types": "crash,osd", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:51:56.895 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:56 smithi164 ceph-mon[83622]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:56.895 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:56 smithi164 ceph-mon[88618]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:57.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:56 smithi172 ceph-mon[67937]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:51:57.559 INFO:teuthology.orchestra.run.smithi164.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:58.256 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-26T02:51:58.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:58.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[88618]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:58.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:58.892 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[88618]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[83622]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:58.893 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:58.894 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:58.894 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:58 smithi164 ceph-mon[83622]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:58 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:51:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:58 smithi172 ceph-mon[67937]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:58 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:58 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:51:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:58 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:51:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:58 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:51:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:58 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:51:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:58 smithi172 ceph-mon[67937]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:51:59.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:51:59 smithi164 ceph-mon[83622]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:51:59.865 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:51:59 smithi164 ceph-mon[88618]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:00.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:51:59 smithi172 ceph-mon[67937]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:00.501 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:52:00.678 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:00.679 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:01.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (6m) 39s ago 28m 26.9M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (6m) 4m ago 28m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (7m) 39s ago 27m 58.5M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (7m) 4m ago 33m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (17m) 39s ago 34m 509M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (4m) 39s ago 34m 49.7M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (5m) 4m ago 33m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:52:01.179 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (4m) 39s ago 33m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (7m) 39s ago 29m 16.2M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (7m) 4m ago 29m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (2m) 39s ago 32m 52.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (95s) 39s ago 32m 44.6M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (44s) 39s ago 32m 17.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (31m) 39s ago 31m 59.6M 2992M 17.2.0 e1d6a67b021e 1e6c0ed88054 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (31m) 4m ago 31m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (30m) 4m ago 30m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (30m) 4m ago 30m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (30m) 4m ago 30m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (6m) 4m ago 29m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (27m) 39s ago 27m 78.9M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:52:01.180 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (27m) 4m ago 27m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:52:01.612 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:52:01.612 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:52:01.612 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:52:01.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:52:01.612 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 8 2024-03-26T02:52:01.613 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:52:01.614 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:52:01.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:52:01.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:52:01.973 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:52:01.973 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:52:01.974 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:52:01.974 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-26T02:52:01.974 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:52:01.974 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "3/8 daemons upgraded", 2024-03-26T02:52:01.974 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:52:01.974 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:52:01.974 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='client.64372 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:02.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:52:02.241 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:52:02] "GET /metrics HTTP/1.1" 200 38185 "" "Prometheus/2.43.0" 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mgr 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mon 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all crash 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:52:03.040 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: Upgrade: osd.3 is safe to restart 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/1224156330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mgr 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mon 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all crash 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: Upgrade: osd.3 is safe to restart 2024-03-26T02:52:03.041 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:02 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/1224156330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:03.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: from='client.54444 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:03.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: from='client.64384 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:03.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:52:03.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:52:03.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mgr 2024-03-26T02:52:03.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mon 2024-03-26T02:52:03.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all crash 2024-03-26T02:52:03.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-03-26T02:52:03.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: Upgrade: osd.3 is safe to restart 2024-03-26T02:52:03.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:02 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/1224156330' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:03.960 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[83622]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:03.960 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[83622]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:03.960 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:03.960 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:52:03.960 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:03.961 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[88618]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:03.961 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[88618]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:03.961 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:03.961 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:52:03.961 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:03 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:04.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:03 smithi172 ceph-mon[67937]: from='client.54462 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:04.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:03 smithi172 ceph-mon[67937]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:04.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:03 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:04.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:03 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T02:52:04.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:03 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:04.841 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:04 smithi164 ceph-mon[88618]: Upgrade: Updating osd.3 2024-03-26T02:52:04.842 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:04 smithi164 ceph-mon[88618]: Deploying daemon osd.3 on smithi164 2024-03-26T02:52:04.842 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:04 smithi164 ceph-mon[83622]: Upgrade: Updating osd.3 2024-03-26T02:52:04.842 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:04 smithi164 ceph-mon[83622]: Deploying daemon osd.3 on smithi164 2024-03-26T02:52:05.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:04 smithi172 ceph-mon[67937]: Upgrade: Updating osd.3 2024-03-26T02:52:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:04 smithi172 ceph-mon[67937]: Deploying daemon osd.3 on smithi164 2024-03-26T02:52:05.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:05 smithi164 ceph-mon[83622]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:05.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:05 smithi164 ceph-mon[88618]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:05 smithi172 ceph-mon[67937]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:06.794 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:06 smithi164 systemd[1]: Stopping Ceph osd.3 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:52:07.044 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:06 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[41283]: 2024-03-26T02:52:06.792+0000 7f6f11c41700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:52:07.044 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:06 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[41283]: 2024-03-26T02:52:06.792+0000 7f6f11c41700 -1 osd.3 127 *** Got signal Terminated *** 2024-03-26T02:52:07.044 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:06 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[41283]: 2024-03-26T02:52:06.792+0000 7f6f11c41700 -1 osd.3 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:52:08.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:07 smithi164 ceph-mon[83622]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:08.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:07 smithi164 ceph-mon[83622]: osd.3 marked itself down and dead 2024-03-26T02:52:08.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:07 smithi164 ceph-mon[88618]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:08.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:07 smithi164 ceph-mon[88618]: osd.3 marked itself down and dead 2024-03-26T02:52:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:07 smithi172 ceph-mon[67937]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:07 smithi172 ceph-mon[67937]: osd.3 marked itself down and dead 2024-03-26T02:52:08.394 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:08 smithi164 podman[113081]: 2024-03-26 02:52:08.111191871 +0000 UTC m=+1.440547403 container died 1e6c0ed8805426dab4157b59fec4fb6c6180fbb76875925eb945b330da6ca4e2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., release=754, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_BRANCH=HEAD, vcs-type=git, architecture=x86_64, io.openshift.expose-services=, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream) 2024-03-26T02:52:08.711 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:08 smithi164 podman[113081]: 2024-03-26 02:52:08.493568399 +0000 UTC m=+1.822923930 container cleanup 1e6c0ed8805426dab4157b59fec4fb6c6180fbb76875925eb945b330da6ca4e2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, name=centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, architecture=x86_64, ceph=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, version=8, io.openshift.expose-services=, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-26T02:52:08.711 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:08 smithi164 bash[113081]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3 2024-03-26T02:52:09.019 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:08 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:09.019 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:08 smithi164 ceph-mon[83622]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:52:09.019 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:08 smithi164 ceph-mon[83622]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T02:52:09.019 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:08 smithi164 ceph-mon[83622]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T02:52:09.020 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:08 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:09.020 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:08 smithi164 ceph-mon[88618]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:52:09.020 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:08 smithi164 ceph-mon[88618]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T02:52:09.020 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:08 smithi164 ceph-mon[88618]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T02:52:09.020 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:08 smithi164 podman[113098]: 2024-03-26 02:52:08.711249155 +0000 UTC m=+0.593057546 container remove 1e6c0ed8805426dab4157b59fec4fb6c6180fbb76875925eb945b330da6ca4e2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, architecture=x86_64, ceph=True, io.buildah.version=1.19.8, release=754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-03-26T02:52:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:08 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:08 smithi172 ceph-mon[67937]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:52:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:08 smithi172 ceph-mon[67937]: osdmap e128: 8 total, 7 up, 8 in 2024-03-26T02:52:09.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:08 smithi172 ceph-mon[67937]: osdmap e129: 8 total, 7 up, 8 in 2024-03-26T02:52:09.294 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:09 smithi164 podman[113187]: 2024-03-26 02:52:08.922251883 +0000 UTC m=+0.018121647 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:52:09.294 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:09 smithi164 podman[113187]: 2024-03-26 02:52:09.040168012 +0000 UTC m=+0.136037775 container create ce37824f5dacaf70f50c65aae178bb2c15b43295eab33d5d28182c257534d2b8 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:52:09.295 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:09 smithi164 podman[113187]: 2024-03-26 02:52:09.264854072 +0000 UTC m=+0.360723836 container init ce37824f5dacaf70f50c65aae178bb2c15b43295eab33d5d28182c257534d2b8 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T02:52:09.295 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:09 smithi164 podman[113187]: 2024-03-26 02:52:09.268168751 +0000 UTC m=+0.364038525 container start ce37824f5dacaf70f50c65aae178bb2c15b43295eab33d5d28182c257534d2b8 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-26T02:52:09.793 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:09 smithi164 podman[113187]: 2024-03-26 02:52:09.356572866 +0000 UTC m=+0.452442639 container attach ce37824f5dacaf70f50c65aae178bb2c15b43295eab33d5d28182c257534d2b8 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T02:52:09.793 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:09 smithi164 podman[113187]: 2024-03-26 02:52:09.556946387 +0000 UTC m=+0.652816144 container died ce37824f5dacaf70f50c65aae178bb2c15b43295eab33d5d28182c257534d2b8 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T02:52:10.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:09 smithi164 ceph-mon[83622]: pgmap v151: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:52:10.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:09 smithi164 ceph-mon[88618]: pgmap v151: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:52:10.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:09 smithi172 ceph-mon[67937]: pgmap v151: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:52:10.544 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 podman[113213]: 2024-03-26 02:52:10.24453959 +0000 UTC m=+0.677908405 container remove ce37824f5dacaf70f50c65aae178bb2c15b43295eab33d5d28182c257534d2b8 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T02:52:10.544 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-26T02:52:10.545 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 systemd[1]: Stopped Ceph osd.3 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:52:10.545 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.3.service: Consumed 17.726s CPU time. 2024-03-26T02:52:10.545 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 systemd[1]: Starting Ceph osd.3 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:52:10.957 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:10 smithi164 ceph-mon[83622]: pgmap v153: 161 pgs: 25 active+undersized+degraded, 1 stale+active+clean, 35 active+undersized, 100 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:52:10.957 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:10 smithi164 ceph-mon[83622]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:10.957 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:10 smithi164 ceph-mon[88618]: pgmap v153: 161 pgs: 25 active+undersized+degraded, 1 stale+active+clean, 35 active+undersized, 100 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:52:10.957 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:10 smithi164 ceph-mon[88618]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:10.958 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 podman[113317]: 2024-03-26 02:52:10.582209032 +0000 UTC m=+0.017640251 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:52:10.958 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 podman[113317]: 2024-03-26 02:52:10.687383208 +0000 UTC m=+0.122814413 container create 7832ee64f4613bed6e0d07b595beeeb0f7437551299db437fb788d28903fa77d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T02:52:11.294 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 podman[113317]: 2024-03-26 02:52:10.956918068 +0000 UTC m=+0.392349274 container init 7832ee64f4613bed6e0d07b595beeeb0f7437551299db437fb788d28903fa77d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T02:52:11.294 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:10 smithi164 podman[113317]: 2024-03-26 02:52:10.960145478 +0000 UTC m=+0.395576692 container start 7832ee64f4613bed6e0d07b595beeeb0f7437551299db437fb788d28903fa77d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T02:52:11.294 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 podman[113317]: 2024-03-26 02:52:11.040031495 +0000 UTC m=+0.475462709 container attach 7832ee64f4613bed6e0d07b595beeeb0f7437551299db437fb788d28903fa77d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:52:11.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:10 smithi172 ceph-mon[67937]: pgmap v153: 161 pgs: 25 active+undersized+degraded, 1 stale+active+clean, 35 active+undersized, 100 active+clean; 585 KiB data, 158 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:52:11.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:10 smithi172 ceph-mon[67937]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 25 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate[113332]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate[113332]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 bash[113317]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 bash[113317]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 bash[113317]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 bash[113317]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 bash[113317]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 bash[113317]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 bash[113317]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T02:52:11.719 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate[113332]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T02:52:11.720 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate[113332]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T02:52:11.720 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate[113332]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-26T02:52:11.720 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate[113332]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-26T02:52:11.720 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate[113332]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-26T02:52:11.720 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:11 smithi164 podman[113317]: 2024-03-26 02:52:11.426466502 +0000 UTC m=+0.861897701 container died 7832ee64f4613bed6e0d07b595beeeb0f7437551299db437fb788d28903fa77d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:52:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:52:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:52:11] "GET /metrics HTTP/1.1" 200 38185 "" "Prometheus/2.43.0" 2024-03-26T02:52:12.294 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:12 smithi164 podman[113468]: 2024-03-26 02:52:12.094388545 +0000 UTC m=+0.661769013 container remove 7832ee64f4613bed6e0d07b595beeeb0f7437551299db437fb788d28903fa77d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T02:52:12.567 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:12 smithi164 podman[113532]: 2024-03-26 02:52:12.218419462 +0000 UTC m=+0.018972651 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:52:12.568 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:12 smithi164 podman[113532]: 2024-03-26 02:52:12.33713619 +0000 UTC m=+0.137689428 container create 724b53d6449c2f4f1d97c56335f2318de80ffeea2a3a76ed538feb232ad5818d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T02:52:12.568 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:12 smithi164 podman[113532]: 2024-03-26 02:52:12.564273828 +0000 UTC m=+0.364827017 container init 724b53d6449c2f4f1d97c56335f2318de80ffeea2a3a76ed538feb232ad5818d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:52:12.568 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:12 smithi164 podman[113532]: 2024-03-26 02:52:12.56738202 +0000 UTC m=+0.367935216 container start 724b53d6449c2f4f1d97c56335f2318de80ffeea2a3a76ed538feb232ad5818d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T02:52:12.857 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:12 smithi164 ceph-osd[113551]: -- 172.21.15.164:0/2573324957 <== mon.0 v2:172.21.15.164:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55baae685680 con 0x55baae64f000 2024-03-26T02:52:12.857 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:12 smithi164 bash[113532]: 724b53d6449c2f4f1d97c56335f2318de80ffeea2a3a76ed538feb232ad5818d 2024-03-26T02:52:12.857 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:12 smithi164 systemd[1]: Started Ceph osd.3 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:52:13.537 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:52:13.108+0000 7fbe9e9fd7c0 -1 Falling back to public interface 2024-03-26T02:52:13.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:13.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:13.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-mon[83622]: pgmap v154: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:13.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:13.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:13.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:13.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-mon[88618]: pgmap v154: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:13.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:13 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:13 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:13 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:13 smithi172 ceph-mon[67937]: pgmap v154: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:14.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:13 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:16.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:15 smithi164 ceph-mon[83622]: pgmap v155: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:16.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:16.041 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:16.042 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:52:15.824+0000 7fbe9e9fd7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-03-26T02:52:16.042 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:52:15.854+0000 7fbe9e9fd7c0 -1 osd.3 127 log_to_monitors true 2024-03-26T02:52:16.042 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:15 smithi164 ceph-mon[88618]: pgmap v155: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:16.042 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:16.042 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:15 smithi172 ceph-mon[67937]: pgmap v155: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:17.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:16 smithi164 ceph-mon[83622]: from='osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:52:17.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:16 smithi164 ceph-mon[83622]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:52:17.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:16 smithi164 ceph-mon[88618]: from='osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:52:17.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:16 smithi164 ceph-mon[88618]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:52:17.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:16 smithi172 ceph-mon[67937]: from='osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:52:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:16 smithi172 ceph-mon[67937]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T02:52:17.544 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:52:17.214+0000 7fbe8be4d700 -1 osd.3 127 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:52:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[88618]: pgmap v156: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[88618]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:52:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[88618]: osdmap e130: 8 total, 7 up, 8 in 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[88618]: from='osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[88618]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[88618]: from='osd.3 ' entity='osd.3' 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[83622]: pgmap v156: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[83622]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[83622]: osdmap e130: 8 total, 7 up, 8 in 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[83622]: from='osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[83622]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:18.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:17 smithi164 ceph-mon[83622]: from='osd.3 ' entity='osd.3' 2024-03-26T02:52:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:17 smithi172 ceph-mon[67937]: pgmap v156: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 162 MiB used, 715 GiB / 715 GiB avail; 96/723 objects degraded (13.278%) 2024-03-26T02:52:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:17 smithi172 ceph-mon[67937]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T02:52:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:17 smithi172 ceph-mon[67937]: osdmap e130: 8 total, 7 up, 8 in 2024-03-26T02:52:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:17 smithi172 ceph-mon[67937]: from='osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:52:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:17 smithi172 ceph-mon[67937]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi164", "root=default"]}]: dispatch 2024-03-26T02:52:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:17 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:18.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:17 smithi172 ceph-mon[67937]: from='osd.3 ' entity='osd.3' 2024-03-26T02:52:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[88618]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[88618]: osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110] boot 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[88618]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[83622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[83622]: osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110] boot 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[83622]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T02:52:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:18 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:52:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:18 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:18 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:18 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:18 smithi172 ceph-mon[67937]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:52:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:18 smithi172 ceph-mon[67937]: osd.3 [v2:172.21.15.164:6826/626109110,v1:172.21.15.164:6827/626109110] boot 2024-03-26T02:52:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:18 smithi172 ceph-mon[67937]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T02:52:19.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:18 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:52:20.028 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:19 smithi164 ceph-mon[83622]: pgmap v159: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 96/723 objects degraded (13.278%) 2024-03-26T02:52:20.028 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:19 smithi164 ceph-mon[83622]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T02:52:20.028 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:19 smithi164 ceph-mon[83622]: Health check update: Degraded data redundancy: 96/723 objects degraded (13.278%), 26 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:20.028 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:19 smithi164 ceph-mon[88618]: pgmap v159: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 96/723 objects degraded (13.278%) 2024-03-26T02:52:20.028 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:19 smithi164 ceph-mon[88618]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T02:52:20.028 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:19 smithi164 ceph-mon[88618]: Health check update: Degraded data redundancy: 96/723 objects degraded (13.278%), 26 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:19 smithi172 ceph-mon[67937]: pgmap v159: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 96/723 objects degraded (13.278%) 2024-03-26T02:52:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:19 smithi172 ceph-mon[67937]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T02:52:20.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:19 smithi172 ceph-mon[67937]: Health check update: Degraded data redundancy: 96/723 objects degraded (13.278%), 26 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:21.999 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:21 smithi164 ceph-mon[83622]: pgmap v161: 161 pgs: 8 peering, 12 active+undersized+degraded, 20 active+undersized, 121 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43/723 objects degraded (5.947%) 2024-03-26T02:52:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:21 smithi164 ceph-mon[88618]: pgmap v161: 161 pgs: 8 peering, 12 active+undersized+degraded, 20 active+undersized, 121 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43/723 objects degraded (5.947%) 2024-03-26T02:52:22.000 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:52:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:52:21.901+0000 7f6ddf0a2700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-26T02:52:22.000 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:52:21 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:52:21] "GET /metrics HTTP/1.1" 200 38192 "" "Prometheus/2.43.0" 2024-03-26T02:52:22.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:21 smithi172 ceph-mon[67937]: pgmap v161: 161 pgs: 8 peering, 12 active+undersized+degraded, 20 active+undersized, 121 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 43/723 objects degraded (5.947%) 2024-03-26T02:52:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:52:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:23.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:23.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[83622]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-26T02:52:24.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[83622]: pgmap v162: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:24.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[83622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/723 objects degraded (5.947%), 12 pgs degraded) 2024-03-26T02:52:24.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[83622]: Cluster is now healthy 2024-03-26T02:52:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[88618]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-26T02:52:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[88618]: pgmap v162: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[88618]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/723 objects degraded (5.947%), 12 pgs degraded) 2024-03-26T02:52:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:23 smithi164 ceph-mon[88618]: Cluster is now healthy 2024-03-26T02:52:24.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:23 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:24.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:23 smithi172 ceph-mon[67937]: Upgrade: unsafe to stop osd(s) at this time (8 PGs are or would become offline) 2024-03-26T02:52:24.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:23 smithi172 ceph-mon[67937]: pgmap v162: 161 pgs: 8 peering, 153 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:24.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:23 smithi172 ceph-mon[67937]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/723 objects degraded (5.947%), 12 pgs degraded) 2024-03-26T02:52:24.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:23 smithi172 ceph-mon[67937]: Cluster is now healthy 2024-03-26T02:52:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:25 smithi164 ceph-mon[83622]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:26.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:25 smithi164 ceph-mon[88618]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:25 smithi172 ceph-mon[67937]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:27 smithi164 ceph-mon[83622]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:28.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:27 smithi164 ceph-mon[88618]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:28.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:27 smithi172 ceph-mon[67937]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:52:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:28 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:28 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:29.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:28 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:29 smithi164 ceph-mon[83622]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-26T02:52:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:29 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:30.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:29 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:29 smithi164 ceph-mon[88618]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-26T02:52:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:29 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:29 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:29 smithi172 ceph-mon[67937]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 0 op/s 2024-03-26T02:52:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:29 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:29 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:31.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:30 smithi164 ceph-mon[83622]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2024-03-26T02:52:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:31.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:30 smithi164 ceph-mon[88618]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2024-03-26T02:52:31.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:31.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:30 smithi172 ceph-mon[67937]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 89 B/s rd, 0 op/s 2024-03-26T02:52:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:52:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:52:32] "GET /metrics HTTP/1.1" 200 38201 "" "Prometheus/2.43.0" 2024-03-26T02:52:32.337 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (7m) 15s ago 29m 26.1M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (6m) 4m ago 28m 81.2M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (8m) 15s ago 28m 58.6M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (8m) 4m ago 33m 448M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (18m) 15s ago 35m 510M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (4m) 15s ago 35m 50.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (5m) 4m ago 33m 40.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (4m) 15s ago 34m 41.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (7m) 15s ago 29m 15.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (7m) 4m ago 29m 15.8M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:52:32.991 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (2m) 15s ago 33m 53.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (2m) 15s ago 32m 45.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (76s) 15s ago 32m 55.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (20s) 15s ago 32m 12.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (31m) 4m ago 31m 56.3M 3248M 17.2.0 e1d6a67b021e a144460139e6 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (31m) 4m ago 31m 52.6M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (31m) 4m ago 30m 55.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (30m) 4m ago 30m 55.7M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (7m) 4m ago 29m 67.8M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (28m) 15s ago 28m 79.2M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:52:32.992 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (28m) 4m ago 28m 80.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:52:33.390 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:52:33.390 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:52:33.390 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:52:33.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:52:33.390 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 4 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 9 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:52:33.391 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "4/8 daemons upgraded", 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:52:33.725 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:52:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:33 smithi164 ceph-mon[83622]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:33 smithi164 ceph-mon[83622]: from='client.54480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:33 smithi164 ceph-mon[83622]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:52:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:33 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/3998323426' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:33 smithi164 ceph-mon[88618]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:33 smithi164 ceph-mon[88618]: from='client.54480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:33 smithi164 ceph-mon[88618]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:52:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:33 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/3998323426' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:34.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:33 smithi172 ceph-mon[67937]: from='client.64408 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:34.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:33 smithi172 ceph-mon[67937]: from='client.54480 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:34.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:33 smithi172 ceph-mon[67937]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:52:34.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:33 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/3998323426' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:34 smithi164 ceph-mon[83622]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:34 smithi164 ceph-mon[83622]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:34 smithi164 ceph-mon[88618]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:34 smithi164 ceph-mon[88618]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:34 smithi172 ceph-mon[67937]: from='client.54486 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:34 smithi172 ceph-mon[67937]: from='client.64432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:52:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:35 smithi164 ceph-mon[83622]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:36.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:35 smithi164 ceph-mon[88618]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:35 smithi172 ceph-mon[67937]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:37 smithi164 ceph-mon[83622]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:37 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:37 smithi164 ceph-mon[88618]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:37 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:37 smithi172 ceph-mon[67937]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:37 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[88618]: Upgrade: osd.4 is safe to restart 2024-03-26T02:52:39.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[83622]: Upgrade: osd.4 is safe to restart 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:52:39.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:38 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:38 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-03-26T02:52:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:38 smithi172 ceph-mon[67937]: Upgrade: osd.4 is safe to restart 2024-03-26T02:52:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:38 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:38 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:38 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T02:52:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:38 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:40.003 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:39 smithi172 ceph-mon[67937]: Upgrade: Updating osd.4 2024-03-26T02:52:40.003 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:39 smithi172 ceph-mon[67937]: Deploying daemon osd.4 on smithi172 2024-03-26T02:52:40.003 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:39 smithi172 ceph-mon[67937]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:39 smithi164 ceph-mon[88618]: Upgrade: Updating osd.4 2024-03-26T02:52:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:39 smithi164 ceph-mon[88618]: Deploying daemon osd.4 on smithi172 2024-03-26T02:52:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:39 smithi164 ceph-mon[88618]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:39 smithi164 ceph-mon[83622]: Upgrade: Updating osd.4 2024-03-26T02:52:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:39 smithi164 ceph-mon[83622]: Deploying daemon osd.4 on smithi172 2024-03-26T02:52:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:39 smithi164 ceph-mon[83622]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:41.929 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:41 smithi172 ceph-mon[67937]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:41.929 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:41 smithi172 systemd[1]: Stopping Ceph osd.4 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:52:42.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:41 smithi164 ceph-mon[83622]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:42.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:41 smithi164 ceph-mon[88618]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:52:42.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:52:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:52:41] "GET /metrics HTTP/1.1" 200 38201 "" "Prometheus/2.43.0" 2024-03-26T02:52:42.306 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[29063]: 2024-03-26T02:52:41.964+0000 7f057fd69700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:52:42.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[29063]: 2024-03-26T02:52:41.964+0000 7f057fd69700 -1 osd.4 132 *** Got signal Terminated *** 2024-03-26T02:52:42.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[29063]: 2024-03-26T02:52:41.964+0000 7f057fd69700 -1 osd.4 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:52:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:42 smithi164 ceph-mon[83622]: osd.4 marked itself down and dead 2024-03-26T02:52:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:42 smithi164 ceph-mon[88618]: osd.4 marked itself down and dead 2024-03-26T02:52:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:42 smithi172 ceph-mon[67937]: osd.4 marked itself down and dead 2024-03-26T02:52:43.532 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:43 smithi172 podman[78389]: 2024-03-26 02:52:43.267589636 +0000 UTC m=+1.415191288 container died a144460139e6c26a148cc627181f659a42627e689c223bf14cde179b71bb34c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_CLEAN=True, vendor=Red Hat, Inc., ceph=True, vcs-type=git, version=8, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.expose-services=, release=754, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870) 2024-03-26T02:52:43.795 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:43 smithi172 podman[78389]: 2024-03-26 02:52:43.623504725 +0000 UTC m=+1.771106389 container cleanup a144460139e6c26a148cc627181f659a42627e689c223bf14cde179b71bb34c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, io.openshift.tags=base centos centos-stream, vcs-type=git, architecture=x86_64, version=8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, RELEASE=HEAD, name=centos-stream, vendor=Red Hat, Inc., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-26T02:52:43.796 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:43 smithi172 bash[78389]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4 2024-03-26T02:52:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:43 smithi172 ceph-mon[67937]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:43 smithi172 ceph-mon[67937]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:52:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:43 smithi172 ceph-mon[67937]: osdmap e133: 8 total, 7 up, 8 in 2024-03-26T02:52:44.057 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:43 smithi172 podman[78418]: 2024-03-26 02:52:43.933583199 +0000 UTC m=+0.669970630 container remove a144460139e6c26a148cc627181f659a42627e689c223bf14cde179b71bb34c0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, io.openshift.expose-services=, version=8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.buildah.version=1.19.8, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64) 2024-03-26T02:52:44.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:43 smithi164 ceph-mon[88618]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:44.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:43 smithi164 ceph-mon[88618]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:52:44.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:43 smithi164 ceph-mon[88618]: osdmap e133: 8 total, 7 up, 8 in 2024-03-26T02:52:44.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:43 smithi164 ceph-mon[83622]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:44.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:43 smithi164 ceph-mon[83622]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:52:44.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:43 smithi164 ceph-mon[83622]: osdmap e133: 8 total, 7 up, 8 in 2024-03-26T02:52:44.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:44 smithi172 podman[78505]: 2024-03-26 02:52:44.284448516 +0000 UTC m=+0.111343695 container create f63b2293ce8c35996658a49820261fe747c62989662fa58e7ecec0056e35dbbd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:52:44.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:44 smithi172 podman[78505]: 2024-03-26 02:52:44.191266946 +0000 UTC m=+0.018162142 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:52:44.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:44 smithi172 podman[78505]: 2024-03-26 02:52:44.511833582 +0000 UTC m=+0.338728761 container init f63b2293ce8c35996658a49820261fe747c62989662fa58e7ecec0056e35dbbd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T02:52:44.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:44 smithi172 podman[78505]: 2024-03-26 02:52:44.515107955 +0000 UTC m=+0.342003132 container start f63b2293ce8c35996658a49820261fe747c62989662fa58e7ecec0056e35dbbd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T02:52:45.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:44 smithi172 ceph-mon[67937]: osdmap e134: 8 total, 7 up, 8 in 2024-03-26T02:52:45.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:44 smithi172 ceph-mon[67937]: pgmap v175: 161 pgs: 5 active+undersized+degraded, 21 stale+active+clean, 4 active+undersized, 131 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 27/723 objects degraded (3.734%) 2024-03-26T02:52:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:45.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:45.058 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:44 smithi172 podman[78505]: 2024-03-26 02:52:44.595374192 +0000 UTC m=+0.422269381 container attach f63b2293ce8c35996658a49820261fe747c62989662fa58e7ecec0056e35dbbd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-03-26T02:52:45.058 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:44 smithi172 podman[78505]: 2024-03-26 02:52:44.806450432 +0000 UTC m=+0.633345627 container died f63b2293ce8c35996658a49820261fe747c62989662fa58e7ecec0056e35dbbd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T02:52:45.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:44 smithi164 ceph-mon[88618]: osdmap e134: 8 total, 7 up, 8 in 2024-03-26T02:52:45.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:44 smithi164 ceph-mon[88618]: pgmap v175: 161 pgs: 5 active+undersized+degraded, 21 stale+active+clean, 4 active+undersized, 131 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 27/723 objects degraded (3.734%) 2024-03-26T02:52:45.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:44 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:45.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:44 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:44 smithi164 ceph-mon[83622]: osdmap e134: 8 total, 7 up, 8 in 2024-03-26T02:52:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:44 smithi164 ceph-mon[83622]: pgmap v175: 161 pgs: 5 active+undersized+degraded, 21 stale+active+clean, 4 active+undersized, 131 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 27/723 objects degraded (3.734%) 2024-03-26T02:52:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:52:45.369 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 podman[78531]: 2024-03-26 02:52:45.3325867 +0000 UTC m=+0.519726134 container remove f63b2293ce8c35996658a49820261fe747c62989662fa58e7ecec0056e35dbbd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T02:52:45.369 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.4.service: Deactivated successfully. 2024-03-26T02:52:45.369 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 systemd[1]: Stopped Ceph osd.4 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:52:45.369 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.4.service: Consumed 17.026s CPU time. 2024-03-26T02:52:45.769 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 systemd[1]: Starting Ceph osd.4 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:52:46.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:45 smithi172 ceph-mon[67937]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:46.045 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 podman[78640]: 2024-03-26 02:52:45.671423795 +0000 UTC m=+0.018019669 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:52:46.045 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 podman[78640]: 2024-03-26 02:52:45.78380124 +0000 UTC m=+0.130397112 container create 376d541e42d8bcc55435b7885f438725f53e74f6569cd179ce9294cad0ad8c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True) 2024-03-26T02:52:46.045 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 podman[78640]: 2024-03-26 02:52:45.969579129 +0000 UTC m=+0.316174993 container init 376d541e42d8bcc55435b7885f438725f53e74f6569cd179ce9294cad0ad8c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T02:52:46.045 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:45 smithi172 podman[78640]: 2024-03-26 02:52:45.972930984 +0000 UTC m=+0.319526856 container start 376d541e42d8bcc55435b7885f438725f53e74f6569cd179ce9294cad0ad8c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:52:46.045 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 podman[78640]: 2024-03-26 02:52:46.044585704 +0000 UTC m=+0.391181576 container attach 376d541e42d8bcc55435b7885f438725f53e74f6569cd179ce9294cad0ad8c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T02:52:46.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:45 smithi164 ceph-mon[83622]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:46.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:45 smithi164 ceph-mon[88618]: Health check failed: Degraded data redundancy: 27/723 objects degraded (3.734%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate[78655]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate[78655]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 bash[78640]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 bash[78640]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 bash[78640]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 bash[78640]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 bash[78640]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 bash[78640]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 bash[78640]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate[78655]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-26T02:52:46.787 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate[78655]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-26T02:52:46.788 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate[78655]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-03-26T02:52:46.788 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate[78655]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-26T02:52:46.788 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate[78655]: --> ceph-volume raw activate successful for osd ID: 4 2024-03-26T02:52:46.788 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 podman[78640]: 2024-03-26 02:52:46.435300018 +0000 UTC m=+0.781895877 container died 376d541e42d8bcc55435b7885f438725f53e74f6569cd179ce9294cad0ad8c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:52:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:46 smithi172 ceph-mon[67937]: pgmap v176: 161 pgs: 25 active+undersized+degraded, 6 stale+active+clean, 34 active+undersized, 96 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 119/723 objects degraded (16.459%) 2024-03-26T02:52:47.057 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:46 smithi172 podman[78782]: 2024-03-26 02:52:46.948090191 +0000 UTC m=+0.507810655 container remove 376d541e42d8bcc55435b7885f438725f53e74f6569cd179ce9294cad0ad8c39 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T02:52:47.217 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:46 smithi164 ceph-mon[83622]: pgmap v176: 161 pgs: 25 active+undersized+degraded, 6 stale+active+clean, 34 active+undersized, 96 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 119/723 objects degraded (16.459%) 2024-03-26T02:52:47.217 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:46 smithi164 ceph-mon[88618]: pgmap v176: 161 pgs: 25 active+undersized+degraded, 6 stale+active+clean, 34 active+undersized, 96 active+clean; 585 KiB data, 179 MiB used, 715 GiB / 715 GiB avail; 119/723 objects degraded (16.459%) 2024-03-26T02:52:47.452 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:47 smithi172 podman[78849]: 2024-03-26 02:52:47.064701347 +0000 UTC m=+0.018311446 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:52:47.452 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:47 smithi172 podman[78849]: 2024-03-26 02:52:47.17422851 +0000 UTC m=+0.127838607 container create 4ca2e0eaa5bbbdc162fe802dca57c2a7a5803fb1c8b921dcf146b4d1db9bbd25 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T02:52:47.452 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:47 smithi172 podman[78849]: 2024-03-26 02:52:47.359904054 +0000 UTC m=+0.313514147 container init 4ca2e0eaa5bbbdc162fe802dca57c2a7a5803fb1c8b921dcf146b4d1db9bbd25 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T02:52:47.452 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:47 smithi172 podman[78849]: 2024-03-26 02:52:47.362942313 +0000 UTC m=+0.316552406 container start 4ca2e0eaa5bbbdc162fe802dca57c2a7a5803fb1c8b921dcf146b4d1db9bbd25 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T02:52:47.452 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:47 smithi172 ceph-osd[78868]: -- 172.21.15.172:0/2605266864 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a47974d680 con 0x55a479717400 2024-03-26T02:52:47.452 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:47 smithi172 bash[78849]: 4ca2e0eaa5bbbdc162fe802dca57c2a7a5803fb1c8b921dcf146b4d1db9bbd25 2024-03-26T02:52:47.806 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:47 smithi172 systemd[1]: Started Ceph osd.4 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:52:48.298 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:47 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:52:47.912+0000 7f907f1557c0 -1 Falling back to public interface 2024-03-26T02:52:48.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:48 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:48.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:48 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:48.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:48 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:48.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:48 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:48 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:48 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:48 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:48.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:48 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:48.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:48 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:48.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:48 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:48.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:48 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:48.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:48 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:49.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:49 smithi164 ceph-mon[83622]: pgmap v177: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:49.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:49 smithi164 ceph-mon[88618]: pgmap v177: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:49.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:49 smithi172 ceph-mon[67937]: pgmap v177: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:50 smithi172 ceph-mon[67937]: from='osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:52:50.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:50 smithi172 ceph-mon[67937]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:52:50.807 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:50 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:52:50.611+0000 7f907f1557c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-03-26T02:52:50.807 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:50 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:52:50.645+0000 7f907f1557c0 -1 osd.4 132 log_to_monitors true 2024-03-26T02:52:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:50 smithi164 ceph-mon[83622]: from='osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:52:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:50 smithi164 ceph-mon[83622]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:52:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:50 smithi164 ceph-mon[88618]: from='osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:52:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:50 smithi164 ceph-mon[88618]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T02:52:51.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:52:51 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:52:51.176+0000 7f906c5a5700 -1 osd.4 132 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:52:52.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[88618]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:52:52.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[88618]: from='osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:52:52.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[88618]: osdmap e135: 8 total, 7 up, 8 in 2024-03-26T02:52:52.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[88618]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:52:52.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[88618]: pgmap v179: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:52.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[88618]: from='osd.4 ' entity='osd.4' 2024-03-26T02:52:52.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[83622]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:52:52.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[83622]: from='osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:52:52.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[83622]: osdmap e135: 8 total, 7 up, 8 in 2024-03-26T02:52:52.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[83622]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:52:52.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[83622]: pgmap v179: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:52.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-mon[83622]: from='osd.4 ' entity='osd.4' 2024-03-26T02:52:52.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:51 smithi172 ceph-mon[67937]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T02:52:52.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:51 smithi172 ceph-mon[67937]: from='osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:52:52.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:51 smithi172 ceph-mon[67937]: osdmap e135: 8 total, 7 up, 8 in 2024-03-26T02:52:52.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:51 smithi172 ceph-mon[67937]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:52:52.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:51 smithi172 ceph-mon[67937]: pgmap v179: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 183 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:52.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:51 smithi172 ceph-mon[67937]: from='osd.4 ' entity='osd.4' 2024-03-26T02:52:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:52:51 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:52:51] "GET /metrics HTTP/1.1" 200 38200 "" "Prometheus/2.43.0" 2024-03-26T02:52:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:52.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[88618]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[88618]: osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074] boot 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[88618]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[83622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[83622]: osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074] boot 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[83622]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T02:52:52.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:52 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:52:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:52 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:53.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:52 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:53.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:52 smithi172 ceph-mon[67937]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:52:53.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:52 smithi172 ceph-mon[67937]: osd.4 [v2:172.21.15.172:6800/4005283074,v1:172.21.15.172:6801/4005283074] boot 2024-03-26T02:52:53.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:52 smithi172 ceph-mon[67937]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T02:52:53.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:52 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:52:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:53 smithi164 ceph-mon[88618]: pgmap v181: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:53 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:53 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:54.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:53 smithi164 ceph-mon[88618]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T02:52:54.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:53 smithi164 ceph-mon[83622]: pgmap v181: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:54.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:53 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:54.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:53 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:54.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:53 smithi164 ceph-mon[83622]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T02:52:54.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:53 smithi172 ceph-mon[67937]: pgmap v181: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 125/723 objects degraded (17.289%) 2024-03-26T02:52:54.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:53 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:54.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:53 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:54.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:53 smithi172 ceph-mon[67937]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T02:52:54.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:54 smithi172 ceph-mon[67937]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:54 smithi164 ceph-mon[83622]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:55.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:54 smithi164 ceph-mon[88618]: Health check update: Degraded data redundancy: 125/723 objects degraded (17.289%), 27 pgs degraded (PG_DEGRADED) 2024-03-26T02:52:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:55 smithi164 ceph-mon[83622]: pgmap v183: 161 pgs: 24 active+undersized+degraded, 38 active+undersized, 99 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/723 objects degraded (14.938%) 2024-03-26T02:52:56.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:55 smithi164 ceph-mon[88618]: pgmap v183: 161 pgs: 24 active+undersized+degraded, 38 active+undersized, 99 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/723 objects degraded (14.938%) 2024-03-26T02:52:56.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:55 smithi172 ceph-mon[67937]: pgmap v183: 161 pgs: 24 active+undersized+degraded, 38 active+undersized, 99 active+clean; 585 KiB data, 194 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 108/723 objects degraded (14.938%) 2024-03-26T02:52:57.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:52:56 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:52:56.584+0000 7f6ddf0a2700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:52:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[88618]: pgmap v184: 161 pgs: 1 active+undersized+degraded, 3 active+undersized, 157 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 5/723 objects degraded (0.692%) 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:52:57.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:52:57.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-26T02:52:57.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:57 smithi164 ceph-mon[83622]: pgmap v184: 161 pgs: 1 active+undersized+degraded, 3 active+undersized, 157 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 5/723 objects degraded (0.692%) 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:52:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:52:57.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:52:57.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: Upgrade: unsafe to stop osd(s) at this time (14 PGs are or would become offline) 2024-03-26T02:52:57.808 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:57 smithi172 ceph-mon[67937]: pgmap v184: 161 pgs: 1 active+undersized+degraded, 3 active+undersized, 157 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 849 B/s rd, 0 op/s; 5/723 objects degraded (0.692%) 2024-03-26T02:52:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:58 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:58 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:58.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:58 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:52:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:52:59 smithi164 ceph-mon[83622]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:52:59 smithi164 ceph-mon[88618]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:52:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:52:59 smithi172 ceph-mon[67937]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:00 smithi164 ceph-mon[83622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/723 objects degraded (0.692%), 1 pg degraded) 2024-03-26T02:53:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:00 smithi164 ceph-mon[83622]: Cluster is now healthy 2024-03-26T02:53:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:00 smithi164 ceph-mon[88618]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/723 objects degraded (0.692%), 1 pg degraded) 2024-03-26T02:53:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:00 smithi164 ceph-mon[88618]: Cluster is now healthy 2024-03-26T02:53:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:00 smithi172 ceph-mon[67937]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 5/723 objects degraded (0.692%), 1 pg degraded) 2024-03-26T02:53:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:00 smithi172 ceph-mon[67937]: Cluster is now healthy 2024-03-26T02:53:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:00.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:01.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:01 smithi164 ceph-mon[83622]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 598 B/s rd, 0 op/s 2024-03-26T02:53:01.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:01 smithi164 ceph-mon[88618]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 598 B/s rd, 0 op/s 2024-03-26T02:53:01.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:01 smithi172 ceph-mon[67937]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 598 B/s rd, 0 op/s 2024-03-26T02:53:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:53:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:53:02] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.43.0" 2024-03-26T02:53:04.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:03 smithi164 ceph-mon[83622]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:04.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:03 smithi164 ceph-mon[88618]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:04.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:03 smithi172 ceph-mon[67937]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:04.089 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:53:04.744 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (7m) 47s ago 29m 26.1M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (7m) 12s ago 29m 81.1M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (8m) 47s ago 28m 58.6M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (8m) 12s ago 34m 453M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (18m) 47s ago 35m 510M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (5m) 47s ago 35m 50.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (6m) 12s ago 34m 50.6M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (5m) 47s ago 34m 41.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (8m) 47s ago 30m 15.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (8m) 12s ago 30m 15.7M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (3m) 47s ago 33m 53.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (2m) 47s ago 33m 45.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (108s) 47s ago 33m 55.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:53:04.745 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (52s) 47s ago 32m 12.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:53:04.746 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (17s) 12s ago 32m 12.6M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:53:04.746 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (31m) 12s ago 31m 57.4M 3248M 17.2.0 e1d6a67b021e 58ac21640a9e 2024-03-26T02:53:04.746 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (31m) 12s ago 31m 58.0M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:53:04.746 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (31m) 12s ago 31m 60.3M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:53:04.746 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (8m) 12s ago 30m 68.6M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:53:04.746 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (28m) 47s ago 28m 79.2M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:53:04.746 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (28m) 12s ago 28m 81.7M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:53:05.145 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:53:05.145 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:53:05.145 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 5 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:53:05.146 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:53:05.147 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-03-26T02:53:05.147 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 10 2024-03-26T02:53:05.147 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:53:05.147 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "5/8 daemons upgraded", 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:53:05.485 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:53:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[88618]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[88618]: from='client.64450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[88618]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-03-26T02:53:06.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[88618]: from='client.54522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:06.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/582465946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[83622]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[83622]: from='client.64450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[83622]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-03-26T02:53:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[83622]: from='client.54522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:06.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:05 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/582465946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:05 smithi172 ceph-mon[67937]: from='client.54510 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:05 smithi172 ceph-mon[67937]: from='client.64450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:05 smithi172 ceph-mon[67937]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 886 B/s rd, 0 op/s 2024-03-26T02:53:06.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:05 smithi172 ceph-mon[67937]: from='client.54522 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:06.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:05 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:06.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:05 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/582465946' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:07.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:06 smithi164 ceph-mon[83622]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:07.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:06 smithi164 ceph-mon[88618]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:07.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:06 smithi172 ceph-mon[67937]: from='client.54534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:08.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:07 smithi172 ceph-mon[67937]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:08.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:07 smithi164 ceph-mon[83622]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:08.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:07 smithi164 ceph-mon[88618]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:08 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:09.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:08 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:09.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:08 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:10.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:09 smithi172 ceph-mon[67937]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:10.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:09 smithi164 ceph-mon[83622]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:10.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:09 smithi164 ceph-mon[88618]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:11.195 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:10 smithi164 ceph-mon[83622]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:11.195 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:10 smithi164 ceph-mon[88618]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:11.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:10 smithi172 ceph-mon[67937]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:12.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:11 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:53:12.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:11 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:53:12.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:11 smithi164 ceph-mon[83622]: Upgrade: osd.5 is safe to restart 2024-03-26T02:53:12.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:11 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:53:12.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:11 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:53:12.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:11 smithi164 ceph-mon[88618]: Upgrade: osd.5 is safe to restart 2024-03-26T02:53:12.209 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:11 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:53:12.209 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:11 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-03-26T02:53:12.209 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:11 smithi172 ceph-mon[67937]: Upgrade: osd.5 is safe to restart 2024-03-26T02:53:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:53:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:53:12] "GET /metrics HTTP/1.1" 200 38204 "" "Prometheus/2.43.0" 2024-03-26T02:53:13.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:12 smithi172 ceph-mon[67937]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:13.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:12 smithi164 ceph-mon[83622]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:13.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:12 smithi164 ceph-mon[88618]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:14.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[88618]: Upgrade: Updating osd.5 2024-03-26T02:53:14.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:14.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:53:14.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:14.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[88618]: Deploying daemon osd.5 on smithi172 2024-03-26T02:53:14.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[83622]: Upgrade: Updating osd.5 2024-03-26T02:53:14.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:14.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:53:14.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:14.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:14 smithi164 ceph-mon[83622]: Deploying daemon osd.5 on smithi172 2024-03-26T02:53:14.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:14 smithi172 ceph-mon[67937]: Upgrade: Updating osd.5 2024-03-26T02:53:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:14 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:14 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T02:53:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:14 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:14.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:14 smithi172 ceph-mon[67937]: Deploying daemon osd.5 on smithi172 2024-03-26T02:53:15.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:15.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:15 smithi172 ceph-mon[67937]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:15.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:15.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:15 smithi164 ceph-mon[83622]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:15.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:15.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:15 smithi164 ceph-mon[88618]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:53:16.553 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:16 smithi172 systemd[1]: Stopping Ceph osd.5 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:53:16.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:16 smithi172 ceph-mon[67937]: osd.5 marked itself down and dead 2024-03-26T02:53:16.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[32014]: 2024-03-26T02:53:16.590+0000 7f5bfd349700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:53:16.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[32014]: 2024-03-26T02:53:16.590+0000 7f5bfd349700 -1 osd.5 137 *** Got signal Terminated *** 2024-03-26T02:53:16.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[32014]: 2024-03-26T02:53:16.590+0000 7f5bfd349700 -1 osd.5 137 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:53:17.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:16 smithi164 ceph-mon[83622]: osd.5 marked itself down and dead 2024-03-26T02:53:17.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:16 smithi164 ceph-mon[88618]: osd.5 marked itself down and dead 2024-03-26T02:53:17.426 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:17 smithi172 podman[83380]: 2024-03-26 02:53:17.153320322 +0000 UTC m=+0.689398199 container died 58ac21640a9e888a47842b27d9956df26a17bb63817b522174452473bd8c6f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, version=8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vendor=Red Hat, Inc., vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, architecture=x86_64, maintainer=Guillaume Abrioux , release=754, ceph=True, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-03-26T02:53:17.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:17 smithi172 podman[83380]: 2024-03-26 02:53:17.515801175 +0000 UTC m=+1.051879034 container cleanup 58ac21640a9e888a47842b27d9956df26a17bb63817b522174452473bd8c6f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, maintainer=Guillaume Abrioux , version=8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vendor=Red Hat, Inc., RELEASE=HEAD, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, release=754, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True) 2024-03-26T02:53:17.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:17 smithi172 bash[83380]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5 2024-03-26T02:53:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:17 smithi172 ceph-mon[67937]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:53:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:17 smithi172 ceph-mon[67937]: osdmap e138: 8 total, 7 up, 8 in 2024-03-26T02:53:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:17 smithi172 ceph-mon[67937]: pgmap v195: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:17.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:17 smithi172 ceph-mon[67937]: osdmap e139: 8 total, 7 up, 8 in 2024-03-26T02:53:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:17 smithi164 ceph-mon[88618]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:53:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:17 smithi164 ceph-mon[88618]: osdmap e138: 8 total, 7 up, 8 in 2024-03-26T02:53:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:17 smithi164 ceph-mon[88618]: pgmap v195: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:18.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:17 smithi164 ceph-mon[88618]: osdmap e139: 8 total, 7 up, 8 in 2024-03-26T02:53:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:17 smithi164 ceph-mon[83622]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:53:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:17 smithi164 ceph-mon[83622]: osdmap e138: 8 total, 7 up, 8 in 2024-03-26T02:53:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:17 smithi164 ceph-mon[83622]: pgmap v195: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:18.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:17 smithi164 ceph-mon[83622]: osdmap e139: 8 total, 7 up, 8 in 2024-03-26T02:53:18.189 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:17 smithi172 podman[83398]: 2024-03-26 02:53:17.833279228 +0000 UTC m=+0.676860206 container remove 58ac21640a9e888a47842b27d9956df26a17bb63817b522174452473bd8c6f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_CLEAN=True, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.tags=base centos centos-stream, ceph=True, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-03-26T02:53:18.474 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:18 smithi172 podman[83494]: 2024-03-26 02:53:18.091974812 +0000 UTC m=+0.020204407 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:53:18.474 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:18 smithi172 podman[83494]: 2024-03-26 02:53:18.210666365 +0000 UTC m=+0.138896009 container create 32ff1257ceb79a798b314f36c1891c1e1e9ae2202bf417bd3c36aac654b61d40 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T02:53:18.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:18 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:18.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:18 smithi172 podman[83494]: 2024-03-26 02:53:18.605330831 +0000 UTC m=+0.533560426 container init 32ff1257ceb79a798b314f36c1891c1e1e9ae2202bf417bd3c36aac654b61d40 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:53:18.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:18 smithi172 podman[83494]: 2024-03-26 02:53:18.608959782 +0000 UTC m=+0.537189371 container start 32ff1257ceb79a798b314f36c1891c1e1e9ae2202bf417bd3c36aac654b61d40 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:53:18.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:18 smithi172 podman[83494]: 2024-03-26 02:53:18.688074308 +0000 UTC m=+0.616303906 container attach 32ff1257ceb79a798b314f36c1891c1e1e9ae2202bf417bd3c36aac654b61d40 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0) 2024-03-26T02:53:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:18 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:18 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:19.202 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:18 smithi172 podman[83494]: 2024-03-26 02:53:18.904479678 +0000 UTC m=+0.832709278 container died 32ff1257ceb79a798b314f36c1891c1e1e9ae2202bf417bd3c36aac654b61d40 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:53:19.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:19 smithi172 ceph-mon[67937]: pgmap v197: 161 pgs: 10 peering, 16 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:53:19.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:19 smithi172 podman[83520]: 2024-03-26 02:53:19.450030519 +0000 UTC m=+0.535454760 container remove 32ff1257ceb79a798b314f36c1891c1e1e9ae2202bf417bd3c36aac654b61d40 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T02:53:19.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:19 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.5.service: Deactivated successfully. 2024-03-26T02:53:19.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:19 smithi172 systemd[1]: Stopped Ceph osd.5 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:53:19.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:19 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.5.service: Consumed 16.343s CPU time. 2024-03-26T02:53:19.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:19 smithi172 systemd[1]: Starting Ceph osd.5 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:53:20.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:19 smithi164 ceph-mon[83622]: pgmap v197: 161 pgs: 10 peering, 16 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:53:20.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:19 smithi164 ceph-mon[88618]: pgmap v197: 161 pgs: 10 peering, 16 stale+active+clean, 135 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:53:20.162 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:19 smithi172 podman[83625]: 2024-03-26 02:53:19.789484769 +0000 UTC m=+0.018033140 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:53:20.162 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:19 smithi172 podman[83625]: 2024-03-26 02:53:19.893036135 +0000 UTC m=+0.121584479 container create 8fde23779420123b723d64e44f23682cd6d0fc18a92bd888370d623811df840d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T02:53:20.557 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 podman[83625]: 2024-03-26 02:53:20.162561913 +0000 UTC m=+0.391110257 container init 8fde23779420123b723d64e44f23682cd6d0fc18a92bd888370d623811df840d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T02:53:20.557 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 podman[83625]: 2024-03-26 02:53:20.165803243 +0000 UTC m=+0.394351594 container start 8fde23779420123b723d64e44f23682cd6d0fc18a92bd888370d623811df840d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T02:53:20.557 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 podman[83625]: 2024-03-26 02:53:20.245913128 +0000 UTC m=+0.474461479 container attach 8fde23779420123b723d64e44f23682cd6d0fc18a92bd888370d623811df840d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T02:53:20.941 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:20 smithi172 ceph-mon[67937]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate[83640]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate[83640]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 bash[83625]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 bash[83625]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 bash[83625]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 bash[83625]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 bash[83625]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 bash[83625]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T02:53:20.941 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 bash[83625]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T02:53:20.942 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate[83640]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T02:53:20.942 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate[83640]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T02:53:20.942 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate[83640]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-26T02:53:20.942 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate[83640]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T02:53:20.942 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate[83640]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T02:53:20.942 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:20 smithi172 podman[83625]: 2024-03-26 02:53:20.634670638 +0000 UTC m=+0.863218993 container died 8fde23779420123b723d64e44f23682cd6d0fc18a92bd888370d623811df840d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T02:53:21.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:20 smithi164 ceph-mon[83622]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-26T02:53:21.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:20 smithi164 ceph-mon[88618]: Health check failed: Reduced data availability: 1 pg inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-26T02:53:21.555 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:21 smithi172 podman[83767]: 2024-03-26 02:53:21.258296747 +0000 UTC m=+0.618463756 container remove 8fde23779420123b723d64e44f23682cd6d0fc18a92bd888370d623811df840d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T02:53:21.555 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:21 smithi172 podman[83832]: 2024-03-26 02:53:21.377570918 +0000 UTC m=+0.018692727 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:53:21.555 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:21 smithi172 podman[83832]: 2024-03-26 02:53:21.475764137 +0000 UTC m=+0.116885931 container create e40eb2ac9a0b8e3a5d5eb2fd063c418885465bb8abdb7605e3c4fbd704e79e27 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T02:53:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:21 smithi172 ceph-mon[67937]: pgmap v198: 161 pgs: 13 active+undersized+degraded, 10 peering, 5 stale+active+clean, 21 active+undersized, 112 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 45/723 objects degraded (6.224%) 2024-03-26T02:53:21.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:21 smithi172 podman[83832]: 2024-03-26 02:53:21.695035013 +0000 UTC m=+0.336156805 container init e40eb2ac9a0b8e3a5d5eb2fd063c418885465bb8abdb7605e3c4fbd704e79e27 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:53:21.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:21 smithi172 podman[83832]: 2024-03-26 02:53:21.699233343 +0000 UTC m=+0.340355129 container start e40eb2ac9a0b8e3a5d5eb2fd063c418885465bb8abdb7605e3c4fbd704e79e27 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T02:53:21.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:21 smithi172 ceph-osd[83851]: -- 172.21.15.172:0/729544404 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56175d1cb680 con 0x56175d195400 2024-03-26T02:53:21.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:21 smithi172 bash[83832]: e40eb2ac9a0b8e3a5d5eb2fd063c418885465bb8abdb7605e3c4fbd704e79e27 2024-03-26T02:53:21.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:21 smithi172 systemd[1]: Started Ceph osd.5 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:53:22.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:21 smithi164 ceph-mon[83622]: pgmap v198: 161 pgs: 13 active+undersized+degraded, 10 peering, 5 stale+active+clean, 21 active+undersized, 112 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 45/723 objects degraded (6.224%) 2024-03-26T02:53:22.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:21 smithi164 ceph-mon[88618]: pgmap v198: 161 pgs: 13 active+undersized+degraded, 10 peering, 5 stale+active+clean, 21 active+undersized, 112 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 45/723 objects degraded (6.224%) 2024-03-26T02:53:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:53:22] "GET /metrics HTTP/1.1" 200 38208 "" "Prometheus/2.43.0" 2024-03-26T02:53:22.557 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:53:22.244+0000 7f49018547c0 -1 Falling back to public interface 2024-03-26T02:53:22.915 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:22 smithi172 ceph-mon[67937]: Health check failed: Degraded data redundancy: 45/723 objects degraded (6.224%), 13 pgs degraded (PG_DEGRADED) 2024-03-26T02:53:22.915 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:22.915 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:22.915 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:23.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-mon[88618]: Health check failed: Degraded data redundancy: 45/723 objects degraded (6.224%), 13 pgs degraded (PG_DEGRADED) 2024-03-26T02:53:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:23.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-mon[83622]: Health check failed: Degraded data redundancy: 45/723 objects degraded (6.224%), 13 pgs degraded (PG_DEGRADED) 2024-03-26T02:53:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:23.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:23.793 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:23 smithi172 ceph-mon[67937]: pgmap v199: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:24.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:23 smithi164 ceph-mon[83622]: pgmap v199: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:24.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:23 smithi164 ceph-mon[88618]: pgmap v199: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:24.958 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:24 smithi172 ceph-mon[67937]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-03-26T02:53:25.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:24 smithi164 ceph-mon[83622]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-03-26T02:53:25.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:24 smithi164 ceph-mon[88618]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 4 pgs peering) 2024-03-26T02:53:25.276 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:53:24.956+0000 7f49018547c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-03-26T02:53:25.277 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:53:24.982+0000 7f49018547c0 -1 osd.5 137 log_to_monitors true 2024-03-26T02:53:26.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:25 smithi164 ceph-mon[83622]: pgmap v200: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:26.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:25 smithi164 ceph-mon[83622]: from='osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:53:26.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:25 smithi164 ceph-mon[83622]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:53:26.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:25 smithi164 ceph-mon[88618]: pgmap v200: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:26.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:25 smithi164 ceph-mon[88618]: from='osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:53:26.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:25 smithi164 ceph-mon[88618]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:53:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:25 smithi172 ceph-mon[67937]: pgmap v200: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 199 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:25 smithi172 ceph-mon[67937]: from='osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:53:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:25 smithi172 ceph-mon[67937]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:53:26.556 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:53:26 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:53:26.172+0000 7f48eeca4700 -1 osd.5 137 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:53:27.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[83622]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[83622]: osdmap e140: 8 total, 7 up, 8 in 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[83622]: from='osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[83622]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[83622]: from='osd.5 ' entity='osd.5' 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[88618]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[88618]: osdmap e140: 8 total, 7 up, 8 in 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[88618]: from='osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:53:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[88618]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:53:27.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[88618]: from='osd.5 ' entity='osd.5' 2024-03-26T02:53:27.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:27.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:26 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:27.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:26 smithi172 ceph-mon[67937]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:53:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:26 smithi172 ceph-mon[67937]: osdmap e140: 8 total, 7 up, 8 in 2024-03-26T02:53:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:26 smithi172 ceph-mon[67937]: from='osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:53:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:26 smithi172 ceph-mon[67937]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:53:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:26 smithi172 ceph-mon[67937]: from='osd.5 ' entity='osd.5' 2024-03-26T02:53:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:27.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:26 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:28.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:27 smithi172 ceph-mon[67937]: pgmap v202: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:28.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:27 smithi172 ceph-mon[67937]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:53:28.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:27 smithi172 ceph-mon[67937]: osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402] boot 2024-03-26T02:53:28.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:27 smithi172 ceph-mon[67937]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T02:53:28.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:53:28.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:28.008 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[83622]: pgmap v202: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[83622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:53:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[83622]: osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402] boot 2024-03-26T02:53:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[83622]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T02:53:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[88618]: pgmap v202: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 82/723 objects degraded (11.342%) 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[88618]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[88618]: osd.5 [v2:172.21.15.172:6808/4045752402,v1:172.21.15.172:6809/4045752402] boot 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[88618]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:28.972 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:28 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:28.972 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:28 smithi172 ceph-mon[67937]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T02:53:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:28 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:28 smithi164 ceph-mon[83622]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T02:53:29.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:28 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:28 smithi164 ceph-mon[88618]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T02:53:30.035 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:29 smithi172 ceph-mon[67937]: pgmap v205: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/723 objects degraded (11.342%) 2024-03-26T02:53:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:29 smithi164 ceph-mon[83622]: pgmap v205: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/723 objects degraded (11.342%) 2024-03-26T02:53:30.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:29 smithi164 ceph-mon[88618]: pgmap v205: 161 pgs: 23 active+undersized+degraded, 36 active+undersized, 102 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 82/723 objects degraded (11.342%) 2024-03-26T02:53:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:30 smithi164 ceph-mon[83622]: Health check update: Degraded data redundancy: 82/723 objects degraded (11.342%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:53:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:31.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:30 smithi164 ceph-mon[88618]: Health check update: Degraded data redundancy: 82/723 objects degraded (11.342%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:53:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:31.045 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:53:30 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:53:30.932+0000 7f6ddf0a2700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T02:53:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:30 smithi172 ceph-mon[67937]: Health check update: Degraded data redundancy: 82/723 objects degraded (11.342%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T02:53:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:32.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:53:31] "GET /metrics HTTP/1.1" 200 38074 "" "Prometheus/2.43.0" 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: pgmap v206: 161 pgs: 13 active+undersized+degraded, 23 active+undersized, 125 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/723 objects degraded (6.224%) 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: pgmap v206: 161 pgs: 13 active+undersized+degraded, 23 active+undersized, 125 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/723 objects degraded (6.224%) 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:32.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: pgmap v206: 161 pgs: 13 active+undersized+degraded, 23 active+undersized, 125 active+clean; 585 KiB data, 214 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 45/723 objects degraded (6.224%) 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:32.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:33.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:32 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:33.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:32 smithi164 ceph-mon[88618]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T02:53:33.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:32 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:33.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:32 smithi164 ceph-mon[83622]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T02:53:33.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:32 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:33.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:32 smithi172 ceph-mon[67937]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-03-26T02:53:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:33 smithi164 ceph-mon[83622]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:53:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:33 smithi164 ceph-mon[83622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/723 objects degraded (6.224%), 13 pgs degraded) 2024-03-26T02:53:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:33 smithi164 ceph-mon[83622]: Cluster is now healthy 2024-03-26T02:53:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:33 smithi164 ceph-mon[88618]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:53:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:33 smithi164 ceph-mon[88618]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/723 objects degraded (6.224%), 13 pgs degraded) 2024-03-26T02:53:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:33 smithi164 ceph-mon[88618]: Cluster is now healthy 2024-03-26T02:53:34.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:33 smithi172 ceph-mon[67937]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:53:34.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:33 smithi172 ceph-mon[67937]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/723 objects degraded (6.224%), 13 pgs degraded) 2024-03-26T02:53:34.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:33 smithi172 ceph-mon[67937]: Cluster is now healthy 2024-03-26T02:53:35.848 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:53:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:35 smithi164 ceph-mon[83622]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:36.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:35 smithi164 ceph-mon[88618]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:35 smithi172 ceph-mon[67937]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:36.503 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:53:36.503 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (8m) 79s ago 30m 26.1M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:53:36.503 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (8m) 10s ago 29m 78.6M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (9m) 79s ago 29m 58.6M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (9m) 10s ago 34m 453M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (19m) 79s ago 36m 510M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (6m) 79s ago 36m 50.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (6m) 10s ago 34m 51.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (5m) 79s ago 35m 41.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (8m) 79s ago 30m 15.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (8m) 10s ago 30m 16.2M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (3m) 79s ago 34m 53.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (3m) 79s ago 33m 45.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (2m) 79s ago 33m 55.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (84s) 79s ago 33m 12.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (49s) 10s ago 32m 54.0M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:53:36.504 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (15s) 10s ago 32m 14.8M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:53:36.505 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (32m) 10s ago 32m 58.6M 3248M 17.2.0 e1d6a67b021e 2f24bcf5dec9 2024-03-26T02:53:36.505 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (31m) 10s ago 31m 60.5M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:53:36.505 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (8m) 10s ago 30m 68.9M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:53:36.505 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (29m) 79s ago 29m 79.2M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:53:36.505 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (29m) 10s ago 29m 81.9M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 6 2024-03-26T02:53:36.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:53:36.903 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:53:36.903 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:53:36.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:53:36.903 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:53:36.903 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-03-26T02:53:36.903 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 11 2024-03-26T02:53:36.903 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:53:36.903 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "6/8 daemons upgraded", 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:53:37.237 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:53:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[83622]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[83622]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[83622]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[83622]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 321 B/s rd, 0 op/s 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/4006523082' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[88618]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[88618]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[88618]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[88618]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 321 B/s rd, 0 op/s 2024-03-26T02:53:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:37 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/4006523082' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:37 smithi172 ceph-mon[67937]: from='client.54543 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:37 smithi172 ceph-mon[67937]: from='client.64483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:37 smithi172 ceph-mon[67937]: from='client.54555 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:37 smithi172 ceph-mon[67937]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 321 B/s rd, 0 op/s 2024-03-26T02:53:38.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:37 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/4006523082' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:53:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:38 smithi172 ceph-mon[67937]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:39.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:38 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:39.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:38 smithi164 ceph-mon[83622]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:39.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:38 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:39.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:38 smithi164 ceph-mon[88618]: from='client.54567 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:53:39.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:38 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:39 smithi172 ceph-mon[67937]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 775 B/s rd, 0 op/s 2024-03-26T02:53:40.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:39 smithi164 ceph-mon[83622]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 775 B/s rd, 0 op/s 2024-03-26T02:53:40.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:39 smithi164 ceph-mon[88618]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 775 B/s rd, 0 op/s 2024-03-26T02:53:41.195 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:41.195 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:40 smithi164 ceph-mon[83622]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:53:41.196 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:41.196 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:40 smithi164 ceph-mon[88618]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:53:41.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:41.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:40 smithi172 ceph-mon[67937]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T02:53:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:53:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:53:41] "GET /metrics HTTP/1.1" 200 38074 "" "Prometheus/2.43.0" 2024-03-26T02:53:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:43 smithi172 ceph-mon[67937]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:53:44.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:43 smithi164 ceph-mon[83622]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:53:44.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:43 smithi164 ceph-mon[88618]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:53:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:46.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:45 smithi164 ceph-mon[88618]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:45 smithi164 ceph-mon[83622]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:53:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:45 smithi172 ceph-mon[67937]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:46.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:46 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:47.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:46 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:47.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:46 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:47.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:47 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:47 smithi172 ceph-mon[67937]: Upgrade: osd.6 is safe to restart 2024-03-26T02:53:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:47 smithi172 ceph-mon[67937]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:47 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:47 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:53:47.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:47 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:48.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:48.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[83622]: Upgrade: osd.6 is safe to restart 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[83622]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[88618]: Upgrade: osd.6 is safe to restart 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[88618]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:53:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:47 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:53:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:48 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:48 smithi164 ceph-mon[83622]: Upgrade: Updating osd.6 2024-03-26T02:53:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:48 smithi164 ceph-mon[83622]: Deploying daemon osd.6 on smithi172 2024-03-26T02:53:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:48 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:48 smithi164 ceph-mon[88618]: Upgrade: Updating osd.6 2024-03-26T02:53:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:48 smithi164 ceph-mon[88618]: Deploying daemon osd.6 on smithi172 2024-03-26T02:53:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:48 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:48 smithi172 ceph-mon[67937]: Upgrade: Updating osd.6 2024-03-26T02:53:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:48 smithi172 ceph-mon[67937]: Deploying daemon osd.6 on smithi172 2024-03-26T02:53:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:49 smithi164 ceph-mon[83622]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:49 smithi164 ceph-mon[88618]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:49 smithi172 ceph-mon[67937]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:53:50.972 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:50 smithi172 systemd[1]: Stopping Ceph osd.6 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:53:51.306 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:50 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[34908]: 2024-03-26T02:53:50.970+0000 7fb3491dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:53:51.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:50 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[34908]: 2024-03-26T02:53:50.970+0000 7fb3491dc700 -1 osd.6 142 *** Got signal Terminated *** 2024-03-26T02:53:51.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:50 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[34908]: 2024-03-26T02:53:50.970+0000 7fb3491dc700 -1 osd.6 142 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:53:52.223 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:51 smithi172 ceph-mon[67937]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:52.223 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:51 smithi172 ceph-mon[67937]: osd.6 marked itself down and dead 2024-03-26T02:53:52.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:51 smithi164 ceph-mon[83622]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:52.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:51 smithi164 ceph-mon[83622]: osd.6 marked itself down and dead 2024-03-26T02:53:52.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:51 smithi164 ceph-mon[88618]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:53:52.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:51 smithi164 ceph-mon[88618]: osd.6 marked itself down and dead 2024-03-26T02:53:52.295 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:53:52] "GET /metrics HTTP/1.1" 200 38218 "" "Prometheus/2.43.0" 2024-03-26T02:53:52.537 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:52 smithi172 podman[87945]: 2024-03-26 02:53:52.223224273 +0000 UTC m=+1.373424202 container died 2f24bcf5dec94f5de048ea89bcef04201373752a776189785ce995981cd71e52 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, release=754, version=8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, architecture=x86_64, RELEASE=HEAD) 2024-03-26T02:53:52.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:52 smithi172 podman[87945]: 2024-03-26 02:53:52.537574049 +0000 UTC m=+1.687773978 container cleanup 2f24bcf5dec94f5de048ea89bcef04201373752a776189785ce995981cd71e52 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, release=754, version=8, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.expose-services=, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-03-26T02:53:52.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:52 smithi172 bash[87945]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6 2024-03-26T02:53:52.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:52 smithi172 podman[87973]: 2024-03-26 02:53:52.763718807 +0000 UTC m=+0.539108153 container remove 2f24bcf5dec94f5de048ea89bcef04201373752a776189785ce995981cd71e52 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-type=git, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, name=centos-stream, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream) 2024-03-26T02:53:53.193 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:52 smithi172 ceph-mon[67937]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:53:53.193 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:52 smithi172 ceph-mon[67937]: osdmap e143: 8 total, 7 up, 8 in 2024-03-26T02:53:53.193 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:52 smithi172 ceph-mon[67937]: pgmap v218: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:53.193 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:52 smithi172 ceph-mon[67937]: osdmap e144: 8 total, 7 up, 8 in 2024-03-26T02:53:53.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-mon[83622]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:53:53.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-mon[83622]: osdmap e143: 8 total, 7 up, 8 in 2024-03-26T02:53:53.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-mon[83622]: pgmap v218: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:53.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-mon[83622]: osdmap e144: 8 total, 7 up, 8 in 2024-03-26T02:53:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-mon[88618]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:53:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-mon[88618]: osdmap e143: 8 total, 7 up, 8 in 2024-03-26T02:53:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-mon[88618]: pgmap v218: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:53:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:52 smithi164 ceph-mon[88618]: osdmap e144: 8 total, 7 up, 8 in 2024-03-26T02:53:53.501 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:53 smithi172 podman[88066]: 2024-03-26 02:53:53.096541548 +0000 UTC m=+0.017915894 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:53:53.501 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:53 smithi172 podman[88066]: 2024-03-26 02:53:53.219050913 +0000 UTC m=+0.140425242 container create 6a7863d0b70af25c127a52d86e09446ca1b32470a3fd3d293f147c0d4a9608a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:53:53.501 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:53 smithi172 podman[88066]: 2024-03-26 02:53:53.434237048 +0000 UTC m=+0.355611378 container init 6a7863d0b70af25c127a52d86e09446ca1b32470a3fd3d293f147c0d4a9608a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T02:53:53.501 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:53 smithi172 podman[88066]: 2024-03-26 02:53:53.43749985 +0000 UTC m=+0.358874187 container start 6a7863d0b70af25c127a52d86e09446ca1b32470a3fd3d293f147c0d4a9608a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:53:53.501 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:53 smithi172 podman[88066]: 2024-03-26 02:53:53.500760425 +0000 UTC m=+0.422134762 container attach 6a7863d0b70af25c127a52d86e09446ca1b32470a3fd3d293f147c0d4a9608a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:53:53.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:53 smithi172 podman[88066]: 2024-03-26 02:53:53.729030562 +0000 UTC m=+0.650404903 container died 6a7863d0b70af25c127a52d86e09446ca1b32470a3fd3d293f147c0d4a9608a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T02:53:54.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 podman[88092]: 2024-03-26 02:53:54.204303549 +0000 UTC m=+0.465999387 container remove 6a7863d0b70af25c127a52d86e09446ca1b32470a3fd3d293f147c0d4a9608a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:53:54.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-26T02:53:54.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 systemd[1]: Stopped Ceph osd.6 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:53:54.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.6.service: Consumed 14.575s CPU time. 2024-03-26T02:53:54.770 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 systemd[1]: Starting Ceph osd.6 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:53:54.770 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 podman[88199]: 2024-03-26 02:53:54.553683108 +0000 UTC m=+0.017945617 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:53:54.770 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 podman[88199]: 2024-03-26 02:53:54.655601268 +0000 UTC m=+0.119863793 container create fa7e658f7df4894ca6805610212979fb8e3e8d95fd5fd33a148cc0a5701c4188 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T02:53:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:54 smithi172 ceph-mon[67937]: pgmap v220: 161 pgs: 1 active+undersized+degraded, 11 stale+active+clean, 2 active+undersized, 147 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/723 objects degraded (0.277%) 2024-03-26T02:53:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:54 smithi172 ceph-mon[67937]: Health check failed: Degraded data redundancy: 2/723 objects degraded (0.277%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:53:55.057 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 podman[88199]: 2024-03-26 02:53:54.850084192 +0000 UTC m=+0.314346677 container init fa7e658f7df4894ca6805610212979fb8e3e8d95fd5fd33a148cc0a5701c4188 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T02:53:55.057 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 podman[88199]: 2024-03-26 02:53:54.853936455 +0000 UTC m=+0.318198937 container start fa7e658f7df4894ca6805610212979fb8e3e8d95fd5fd33a148cc0a5701c4188 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T02:53:55.057 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:54 smithi172 podman[88199]: 2024-03-26 02:53:54.950085291 +0000 UTC m=+0.414347792 container attach fa7e658f7df4894ca6805610212979fb8e3e8d95fd5fd33a148cc0a5701c4188 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD) 2024-03-26T02:53:55.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:54 smithi164 ceph-mon[83622]: pgmap v220: 161 pgs: 1 active+undersized+degraded, 11 stale+active+clean, 2 active+undersized, 147 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/723 objects degraded (0.277%) 2024-03-26T02:53:55.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:54 smithi164 ceph-mon[83622]: Health check failed: Degraded data redundancy: 2/723 objects degraded (0.277%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:53:55.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:54 smithi164 ceph-mon[88618]: pgmap v220: 161 pgs: 1 active+undersized+degraded, 11 stale+active+clean, 2 active+undersized, 147 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 2/723 objects degraded (0.277%) 2024-03-26T02:53:55.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:54 smithi164 ceph-mon[88618]: Health check failed: Degraded data redundancy: 2/723 objects degraded (0.277%), 1 pg degraded (PG_DEGRADED) 2024-03-26T02:53:55.557 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate[88213]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T02:53:55.557 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate[88213]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T02:53:55.557 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 bash[88199]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 bash[88199]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 bash[88199]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 bash[88199]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 bash[88199]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 bash[88199]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 bash[88199]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate[88213]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate[88213]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate[88213]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate[88213]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate[88213]: --> ceph-volume raw activate successful for osd ID: 6 2024-03-26T02:53:55.558 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 podman[88199]: 2024-03-26 02:53:55.324031231 +0000 UTC m=+0.788293724 container died fa7e658f7df4894ca6805610212979fb8e3e8d95fd5fd33a148cc0a5701c4188 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T02:53:56.056 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:55 smithi172 podman[88347]: 2024-03-26 02:53:55.895393714 +0000 UTC m=+0.562513726 container remove fa7e658f7df4894ca6805610212979fb8e3e8d95fd5fd33a148cc0a5701c4188 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-activate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:53:56.361 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:56 smithi172 podman[88414]: 2024-03-26 02:53:56.011511486 +0000 UTC m=+0.018184124 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:53:56.361 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:56 smithi172 podman[88414]: 2024-03-26 02:53:56.121423475 +0000 UTC m=+0.128096085 container create 44e8c5cf078f8fae118fdf3d76a92be3e9c9c4cfa30eb9bae0d4a3a91d3a8e42 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:53:56.361 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:56 smithi172 podman[88414]: 2024-03-26 02:53:56.357804581 +0000 UTC m=+0.364477191 container init 44e8c5cf078f8fae118fdf3d76a92be3e9c9c4cfa30eb9bae0d4a3a91d3a8e42 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T02:53:56.361 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:56 smithi172 podman[88414]: 2024-03-26 02:53:56.360865165 +0000 UTC m=+0.367537785 container start 44e8c5cf078f8fae118fdf3d76a92be3e9c9c4cfa30eb9bae0d4a3a91d3a8e42 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:53:56.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:56 smithi172 ceph-osd[88432]: -- 172.21.15.172:0/393433851 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558ed9311680 con 0x558ed92db400 2024-03-26T02:53:56.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:56 smithi172 bash[88414]: 44e8c5cf078f8fae118fdf3d76a92be3e9c9c4cfa30eb9bae0d4a3a91d3a8e42 2024-03-26T02:53:56.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:56 smithi172 systemd[1]: Started Ceph osd.6 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:53:57.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:56 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:53:56.900+0000 7f7c310fb7c0 -1 Falling back to public interface 2024-03-26T02:53:57.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:57.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:57.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:57 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:57.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:57 smithi164 ceph-mon[83622]: pgmap v221: 161 pgs: 17 active+undersized+degraded, 1 stale+active+clean, 25 active+undersized, 118 active+clean; 585 KiB data, 219 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-03-26T02:53:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:57 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:57.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:57 smithi164 ceph-mon[88618]: pgmap v221: 161 pgs: 17 active+undersized+degraded, 1 stale+active+clean, 25 active+undersized, 118 active+clean; 585 KiB data, 219 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-03-26T02:53:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:53:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:57 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:53:57.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:57 smithi172 ceph-mon[67937]: pgmap v221: 161 pgs: 17 active+undersized+degraded, 1 stale+active+clean, 25 active+undersized, 118 active+clean; 585 KiB data, 219 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-03-26T02:53:58.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:58 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:58.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:58 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:58.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:58 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:53:59.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:53:59 smithi164 ceph-mon[83622]: pgmap v222: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 219 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:53:59.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:53:59 smithi164 ceph-mon[88618]: pgmap v222: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 219 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:53:59.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:53:59 smithi172 ceph-mon[67937]: pgmap v222: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 219 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:53:59.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:53:59.626+0000 7f7c310fb7c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-03-26T02:53:59.807 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:53:59 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:53:59.653+0000 7f7c310fb7c0 -1 osd.6 142 log_to_monitors true 2024-03-26T02:54:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:00 smithi164 ceph-mon[83622]: from='osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:54:00.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:00 smithi164 ceph-mon[83622]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:54:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:00.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:00 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:00 smithi164 ceph-mon[88618]: from='osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:54:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:00 smithi164 ceph-mon[88618]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:54:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:00.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:00 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:00 smithi172 ceph-mon[67937]: from='osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:54:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:00 smithi172 ceph-mon[67937]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:54:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:00.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:00 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:01.306 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:54:01.017+0000 7f7c1e54b700 -1 osd.6 142 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:54:01.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[83622]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[83622]: osdmap e145: 8 total, 7 up, 8 in 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[83622]: from='osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[83622]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[83622]: pgmap v224: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[83622]: from='osd.6 ' entity='osd.6' 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[88618]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[88618]: osdmap e145: 8 total, 7 up, 8 in 2024-03-26T02:54:01.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[88618]: from='osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[88618]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[88618]: pgmap v224: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:54:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:01.796 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:01 smithi164 ceph-mon[88618]: from='osd.6 ' entity='osd.6' 2024-03-26T02:54:01.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-mon[67937]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:54:01.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-mon[67937]: osdmap e145: 8 total, 7 up, 8 in 2024-03-26T02:54:01.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-mon[67937]: from='osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:01.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-mon[67937]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:01.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:01.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-mon[67937]: pgmap v224: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:54:01.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:01.807 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:01 smithi172 ceph-mon[67937]: from='osd.6 ' entity='osd.6' 2024-03-26T02:54:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:54:02] "GET /metrics HTTP/1.1" 200 38225 "" "Prometheus/2.43.0" 2024-03-26T02:54:03.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:03.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[88618]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[88618]: osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779] boot 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[88618]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[88618]: pgmap v226: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[83622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[83622]: osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779] boot 2024-03-26T02:54:03.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[83622]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T02:54:03.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:54:03.296 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:02 smithi164 ceph-mon[83622]: pgmap v226: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:54:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:02 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:02 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:02 smithi172 ceph-mon[67937]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:54:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:02 smithi172 ceph-mon[67937]: osd.6 [v2:172.21.15.172:6816/3034840779,v1:172.21.15.172:6817/3034840779] boot 2024-03-26T02:54:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:02 smithi172 ceph-mon[67937]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T02:54:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:02 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:54:03.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:02 smithi172 ceph-mon[67937]: pgmap v226: 161 pgs: 22 active+undersized+degraded, 31 active+undersized, 108 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 89/723 objects degraded (12.310%) 2024-03-26T02:54:04.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:04 smithi164 ceph-mon[83622]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T02:54:04.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:04 smithi164 ceph-mon[88618]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T02:54:04.301 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:04 smithi172 ceph-mon[67937]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T02:54:05.173 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:05 smithi172 ceph-mon[67937]: Health check update: Degraded data redundancy: 89/723 objects degraded (12.310%), 22 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:05.173 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:05 smithi172 ceph-mon[67937]: pgmap v228: 161 pgs: 22 active+undersized+degraded, 30 active+undersized, 109 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/723 objects degraded (12.310%) 2024-03-26T02:54:05.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:05 smithi164 ceph-mon[83622]: Health check update: Degraded data redundancy: 89/723 objects degraded (12.310%), 22 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:05.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:05 smithi164 ceph-mon[83622]: pgmap v228: 161 pgs: 22 active+undersized+degraded, 30 active+undersized, 109 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/723 objects degraded (12.310%) 2024-03-26T02:54:05.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:05 smithi164 ceph-mon[88618]: Health check update: Degraded data redundancy: 89/723 objects degraded (12.310%), 22 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:05.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:05 smithi164 ceph-mon[88618]: pgmap v228: 161 pgs: 22 active+undersized+degraded, 30 active+undersized, 109 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 89/723 objects degraded (12.310%) 2024-03-26T02:54:05.794 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:54:05 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:54:05.405+0000 7f6ddf0a2700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-26T02:54:06.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:06.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[88618]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.546 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:06.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:06.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:06 smithi164 ceph-mon[83622]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:06.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:06.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:06.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:06 smithi172 ceph-mon[67937]: Upgrade: unsafe to stop osd(s) at this time (19 PGs are or would become offline) 2024-03-26T02:54:07.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:07 smithi164 ceph-mon[83622]: pgmap v229: 161 pgs: 4 active+undersized+degraded, 6 active+undersized, 151 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 14/723 objects degraded (1.936%) 2024-03-26T02:54:07.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:07 smithi164 ceph-mon[88618]: pgmap v229: 161 pgs: 4 active+undersized+degraded, 6 active+undersized, 151 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 14/723 objects degraded (1.936%) 2024-03-26T02:54:07.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:07 smithi172 ceph-mon[67937]: pgmap v229: 161 pgs: 4 active+undersized+degraded, 6 active+undersized, 151 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 14/723 objects degraded (1.936%) 2024-03-26T02:54:07.598 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:54:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:54:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (8m) 111s ago 30m 26.1M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:54:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (8m) 7s ago 30m 80.5M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:54:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (9m) 111s ago 29m 58.6M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:54:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (9m) 7s ago 35m 453M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:54:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (19m) 111s ago 36m 510M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:54:08.264 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (6m) 111s ago 36m 50.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (7m) 7s ago 35m 53.2M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (6m) 111s ago 35m 41.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (9m) 111s ago 31m 15.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (9m) 7s ago 31m 16.5M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (4m) 111s ago 34m 53.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (3m) 111s ago 34m 45.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (2m) 111s ago 34m 55.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (116s) 111s ago 33m 12.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (81s) 7s ago 33m 54.3M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (46s) 7s ago 32m 53.6M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (12s) 7s ago 32m 11.7M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 44e8c5cf078f 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (32m) 7s ago 32m 61.5M 3248M 17.2.0 e1d6a67b021e 839f52afbd56 2024-03-26T02:54:08.265 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (9m) 7s ago 31m 75.0M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:54:08.266 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (29m) 111s ago 29m 79.2M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:54:08.266 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (29m) 7s ago 29m 82.1M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:54:08.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:08 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:08.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:08 smithi164 ceph-mon[83622]: from='client.54576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:08.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:08 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:08.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:08 smithi164 ceph-mon[88618]: from='client.54576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:08.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:08 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:08.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:08 smithi172 ceph-mon[67937]: from='client.54576 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:08.661 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:54:08.661 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:54:08.661 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:54:08.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 7 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:54:08.662 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-03-26T02:54:08.663 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 12 2024-03-26T02:54:08.663 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:54:08.663 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:54:08.999 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:54:08.999 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:54:08.999 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:54:08.999 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-26T02:54:08.999 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:54:09.000 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "7/8 daemons upgraded", 2024-03-26T02:54:09.000 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:54:09.000 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:54:09.000 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:54:09.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:09 smithi164 ceph-mon[83622]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:09.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:09 smithi164 ceph-mon[83622]: from='client.54588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:09.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:09 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/418597031' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:09.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:09 smithi164 ceph-mon[83622]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:54:09.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:09 smithi164 ceph-mon[88618]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:09.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:09 smithi164 ceph-mon[88618]: from='client.54588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:09.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:09 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/418597031' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:09.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:09 smithi164 ceph-mon[88618]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:54:09.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:09 smithi172 ceph-mon[67937]: from='client.54582 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:09.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:09 smithi172 ceph-mon[67937]: from='client.54588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:09.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:09 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/418597031' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:09.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:09 smithi172 ceph-mon[67937]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:54:10.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:10 smithi164 ceph-mon[83622]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:10.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:10 smithi164 ceph-mon[83622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/723 objects degraded (1.936%), 4 pgs degraded) 2024-03-26T02:54:10.545 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:10 smithi164 ceph-mon[83622]: Cluster is now healthy 2024-03-26T02:54:10.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:10 smithi164 ceph-mon[88618]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:10.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:10 smithi164 ceph-mon[88618]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/723 objects degraded (1.936%), 4 pgs degraded) 2024-03-26T02:54:10.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:10 smithi164 ceph-mon[88618]: Cluster is now healthy 2024-03-26T02:54:10.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:10 smithi172 ceph-mon[67937]: from='client.64534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:10.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:10 smithi172 ceph-mon[67937]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/723 objects degraded (1.936%), 4 pgs degraded) 2024-03-26T02:54:10.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:10 smithi172 ceph-mon[67937]: Cluster is now healthy 2024-03-26T02:54:11.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:11 smithi164 ceph-mon[83622]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:54:11.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:11 smithi164 ceph-mon[88618]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:54:11.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:11 smithi172 ceph-mon[67937]: pgmap v231: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 937 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:54:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:54:11 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:54:11] "GET /metrics HTTP/1.1" 200 38225 "" "Prometheus/2.43.0" 2024-03-26T02:54:14.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:13 smithi164 ceph-mon[83622]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:54:14.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:13 smithi164 ceph-mon[88618]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:54:14.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:13 smithi172 ceph-mon[67937]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 0 B/s wr, 16 op/s 2024-03-26T02:54:16.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:16.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:16.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:15 smithi164 ceph-mon[83622]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-03-26T02:54:16.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:15 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:15 smithi164 ceph-mon[88618]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-03-26T02:54:16.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:15 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:15 smithi172 ceph-mon[67937]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 34 op/s 2024-03-26T02:54:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:15 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:17.293 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:16 smithi164 ceph-mon[83622]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-03-26T02:54:17.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:16 smithi164 ceph-mon[88618]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-03-26T02:54:17.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:16 smithi172 ceph-mon[67937]: pgmap v234: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 0 B/s wr, 109 op/s 2024-03-26T02:54:18.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:17 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:18.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:17 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:18.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:17 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:19.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:18 smithi164 ceph-mon[83622]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-26T02:54:19.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:18 smithi164 ceph-mon[88618]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-26T02:54:19.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:18 smithi172 ceph-mon[67937]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-03-26T02:54:20.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:20 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:20.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:20 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:20 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:21.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:21 smithi172 ceph-mon[67937]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:21.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:21 smithi172 ceph-mon[67937]: Upgrade: osd.7 is safe to restart 2024-03-26T02:54:21.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:21 smithi172 ceph-mon[67937]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:54:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:21 smithi164 ceph-mon[83622]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:21 smithi164 ceph-mon[83622]: Upgrade: osd.7 is safe to restart 2024-03-26T02:54:21.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:21 smithi164 ceph-mon[83622]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:54:21.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:21 smithi164 ceph-mon[88618]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-03-26T02:54:21.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:21 smithi164 ceph-mon[88618]: Upgrade: osd.7 is safe to restart 2024-03-26T02:54:21.795 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:21 smithi164 ceph-mon[88618]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:54:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:54:21] "GET /metrics HTTP/1.1" 200 38230 "" "Prometheus/2.43.0" 2024-03-26T02:54:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:22 smithi172 ceph-mon[67937]: Upgrade: Updating osd.7 2024-03-26T02:54:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:54:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:22 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:22 smithi172 ceph-mon[67937]: Deploying daemon osd.7 on smithi172 2024-03-26T02:54:23.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:22 smithi172 ceph-mon[67937]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:54:23.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[83622]: Upgrade: Updating osd.7 2024-03-26T02:54:23.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[83622]: Deploying daemon osd.7 on smithi172 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[83622]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[88618]: Upgrade: Updating osd.7 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[88618]: Deploying daemon osd.7 on smithi172 2024-03-26T02:54:23.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:22 smithi164 ceph-mon[88618]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-03-26T02:54:25.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:24 smithi172 ceph-mon[67937]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T02:54:25.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:24 smithi164 ceph-mon[83622]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T02:54:25.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:24 smithi164 ceph-mon[88618]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 64 KiB/s rd, 0 B/s wr, 106 op/s 2024-03-26T02:54:25.359 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:25 smithi172 systemd[1]: Stopping Ceph osd.7 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:54:25.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[37834]: 2024-03-26T02:54:25.361+0000 7f65d0787700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:54:25.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[37834]: 2024-03-26T02:54:25.361+0000 7f65d0787700 -1 osd.7 147 *** Got signal Terminated *** 2024-03-26T02:54:25.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[37834]: 2024-03-26T02:54:25.361+0000 7f65d0787700 -1 osd.7 147 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:54:26.210 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:25 smithi172 ceph-mon[67937]: osd.7 marked itself down and dead 2024-03-26T02:54:26.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:25 smithi164 ceph-mon[83622]: osd.7 marked itself down and dead 2024-03-26T02:54:26.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:25 smithi164 ceph-mon[88618]: osd.7 marked itself down and dead 2024-03-26T02:54:26.521 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:26 smithi172 podman[92965]: 2024-03-26 02:54:26.210124384 +0000 UTC m=+0.950539469 container died 839f52afbd56acec707da6ec1ed53077af330b1ab3452eaaefebcb9c4d297442 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, version=8, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux ) 2024-03-26T02:54:26.806 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:26 smithi172 podman[92965]: 2024-03-26 02:54:26.620240949 +0000 UTC m=+1.360656014 container cleanup 839f52afbd56acec707da6ec1ed53077af330b1ab3452eaaefebcb9c4d297442 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.component=centos-stream-container, name=centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, version=8, GIT_BRANCH=HEAD, io.openshift.expose-services=) 2024-03-26T02:54:26.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:26 smithi172 bash[92965]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7 2024-03-26T02:54:27.183 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:26 smithi172 ceph-mon[67937]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:54:27.183 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:26 smithi172 ceph-mon[67937]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T02:54:27.183 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:26 smithi172 ceph-mon[67937]: osdmap e148: 8 total, 7 up, 8 in 2024-03-26T02:54:27.183 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:26 smithi172 ceph-mon[67937]: pgmap v240: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:54:27.183 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:26 smithi172 podman[92983]: 2024-03-26 02:54:26.854918171 +0000 UTC m=+0.641943248 container remove 839f52afbd56acec707da6ec1ed53077af330b1ab3452eaaefebcb9c4d297442 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, name=centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD) 2024-03-26T02:54:27.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:26 smithi164 ceph-mon[83622]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:54:27.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:26 smithi164 ceph-mon[83622]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T02:54:27.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:26 smithi164 ceph-mon[83622]: osdmap e148: 8 total, 7 up, 8 in 2024-03-26T02:54:27.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:26 smithi164 ceph-mon[83622]: pgmap v240: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:54:27.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:26 smithi164 ceph-mon[88618]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T02:54:27.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:26 smithi164 ceph-mon[88618]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-03-26T02:54:27.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:26 smithi164 ceph-mon[88618]: osdmap e148: 8 total, 7 up, 8 in 2024-03-26T02:54:27.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:26 smithi164 ceph-mon[88618]: pgmap v240: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 8.2 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-26T02:54:27.516 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:27 smithi172 podman[93080]: 2024-03-26 02:54:27.086053888 +0000 UTC m=+0.020829365 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:54:27.517 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:27 smithi172 podman[93080]: 2024-03-26 02:54:27.210729229 +0000 UTC m=+0.145504691 container create 2d5d4fa1b7473f41aa857314f022dd050a786b0238d63040245515ac6dcac883 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:54:27.517 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:27 smithi172 podman[93080]: 2024-03-26 02:54:27.516602436 +0000 UTC m=+0.451377898 container init 2d5d4fa1b7473f41aa857314f022dd050a786b0238d63040245515ac6dcac883 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T02:54:27.806 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:27 smithi172 podman[93080]: 2024-03-26 02:54:27.519761091 +0000 UTC m=+0.454536560 container start 2d5d4fa1b7473f41aa857314f022dd050a786b0238d63040245515ac6dcac883 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:54:27.806 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:27 smithi172 podman[93080]: 2024-03-26 02:54:27.608390694 +0000 UTC m=+0.543166167 container attach 2d5d4fa1b7473f41aa857314f022dd050a786b0238d63040245515ac6dcac883 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:54:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:27 smithi172 ceph-mon[67937]: osdmap e149: 8 total, 7 up, 8 in 2024-03-26T02:54:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:27 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:28.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:27 smithi172 podman[93080]: 2024-03-26 02:54:27.806149335 +0000 UTC m=+0.740924802 container died 2d5d4fa1b7473f41aa857314f022dd050a786b0238d63040245515ac6dcac883 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:54:28.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:27 smithi164 ceph-mon[83622]: osdmap e149: 8 total, 7 up, 8 in 2024-03-26T02:54:28.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:27 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:28.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:27 smithi164 ceph-mon[88618]: osdmap e149: 8 total, 7 up, 8 in 2024-03-26T02:54:28.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:27 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:28.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:28 smithi172 podman[93106]: 2024-03-26 02:54:28.503779469 +0000 UTC m=+0.691656198 container remove 2d5d4fa1b7473f41aa857314f022dd050a786b0238d63040245515ac6dcac883 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T02:54:28.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:28 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.7.service: Deactivated successfully. 2024-03-26T02:54:28.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:28 smithi172 systemd[1]: Stopped Ceph osd.7 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:54:28.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:28 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.7.service: Consumed 17.190s CPU time. 2024-03-26T02:54:28.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:28 smithi172 systemd[1]: Starting Ceph osd.7 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:54:29.157 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:28 smithi172 ceph-mon[67937]: pgmap v242: 161 pgs: 13 peering, 22 stale+active+clean, 126 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:54:29.158 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:28 smithi172 podman[93209]: 2024-03-26 02:54:28.946548112 +0000 UTC m=+0.113870700 container create 6b1a1b77442bd566c31ff168a28b1d8a1909bf4a73174c5d766445d488e164d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:54:29.158 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:28 smithi172 podman[93209]: 2024-03-26 02:54:28.850653023 +0000 UTC m=+0.017975624 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:54:29.158 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 podman[93209]: 2024-03-26 02:54:29.157518409 +0000 UTC m=+0.324840996 container init 6b1a1b77442bd566c31ff168a28b1d8a1909bf4a73174c5d766445d488e164d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T02:54:29.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:28 smithi164 ceph-mon[83622]: pgmap v242: 161 pgs: 13 peering, 22 stale+active+clean, 126 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:54:29.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:28 smithi164 ceph-mon[88618]: pgmap v242: 161 pgs: 13 peering, 22 stale+active+clean, 126 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:54:29.557 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 podman[93209]: 2024-03-26 02:54:29.160743148 +0000 UTC m=+0.328065745 container start 6b1a1b77442bd566c31ff168a28b1d8a1909bf4a73174c5d766445d488e164d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:54:29.557 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 podman[93209]: 2024-03-26 02:54:29.24079593 +0000 UTC m=+0.408118532 container attach 6b1a1b77442bd566c31ff168a28b1d8a1909bf4a73174c5d766445d488e164d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-26T02:54:30.012 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate[93224]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T02:54:30.012 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate[93224]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T02:54:30.012 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 bash[93209]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T02:54:30.012 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 bash[93209]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-26T02:54:30.012 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 bash[93209]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 bash[93209]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 bash[93209]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 bash[93209]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 bash[93209]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate[93224]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate[93224]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate[93224]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate[93224]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate[93224]: --> ceph-volume raw activate successful for osd ID: 7 2024-03-26T02:54:30.013 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:29 smithi172 podman[93209]: 2024-03-26 02:54:29.624976898 +0000 UTC m=+0.792299478 container died 6b1a1b77442bd566c31ff168a28b1d8a1909bf4a73174c5d766445d488e164d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:54:30.556 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:30 smithi172 podman[93346]: 2024-03-26 02:54:30.294984643 +0000 UTC m=+0.664677783 container remove 6b1a1b77442bd566c31ff168a28b1d8a1909bf4a73174c5d766445d488e164d9 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T02:54:30.557 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:30 smithi172 podman[93413]: 2024-03-26 02:54:30.418326469 +0000 UTC m=+0.018239294 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:54:30.557 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:30 smithi172 podman[93413]: 2024-03-26 02:54:30.529250455 +0000 UTC m=+0.129163251 container create 66123d33bbf266b1cac6d6fbed4910acf490794c35ffa8ece9ea726c03f7fc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:54:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:31.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:31.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:31.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:30 smithi172 podman[93413]: 2024-03-26 02:54:30.757096301 +0000 UTC m=+0.357009111 container init 66123d33bbf266b1cac6d6fbed4910acf490794c35ffa8ece9ea726c03f7fc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:54:31.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:30 smithi172 podman[93413]: 2024-03-26 02:54:30.760238411 +0000 UTC m=+0.360151214 container start 66123d33bbf266b1cac6d6fbed4910acf490794c35ffa8ece9ea726c03f7fc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T02:54:31.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:30 smithi172 ceph-osd[93432]: -- 172.21.15.172:0/1843913302 <== mon.2 v2:172.21.15.172:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612007b5680 con 0x56120077f400 2024-03-26T02:54:31.058 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:30 smithi172 bash[93413]: 66123d33bbf266b1cac6d6fbed4910acf490794c35ffa8ece9ea726c03f7fc34 2024-03-26T02:54:31.058 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:30 smithi172 systemd[1]: Started Ceph osd.7 for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:54:31.735 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:54:31.306+0000 7fe9accf07c0 -1 Falling back to public interface 2024-03-26T02:54:32.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:31 smithi164 ceph-mon[83622]: pgmap v243: 161 pgs: 8 active+undersized+degraded, 13 peering, 13 stale+active+clean, 20 active+undersized, 107 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-03-26T02:54:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:31 smithi164 ceph-mon[88618]: pgmap v243: 161 pgs: 8 active+undersized+degraded, 13 peering, 13 stale+active+clean, 20 active+undersized, 107 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-03-26T02:54:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:32.043 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:31 smithi172 ceph-mon[67937]: pgmap v243: 161 pgs: 8 active+undersized+degraded, 13 peering, 13 stale+active+clean, 20 active+undersized, 107 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 16/723 objects degraded (2.213%) 2024-03-26T02:54:32.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:32.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:32.044 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:54:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:54:32] "GET /metrics HTTP/1.1" 200 38306 "" "Prometheus/2.43.0" 2024-03-26T02:54:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:32 smithi164 ceph-mon[83622]: Health check failed: Degraded data redundancy: 16/723 objects degraded (2.213%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:33.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:32 smithi164 ceph-mon[88618]: Health check failed: Degraded data redundancy: 16/723 objects degraded (2.213%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:33.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:32 smithi172 ceph-mon[67937]: Health check failed: Degraded data redundancy: 16/723 objects degraded (2.213%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:34.033 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:33 smithi172 ceph-mon[67937]: pgmap v244: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:54:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:33 smithi164 ceph-mon[83622]: pgmap v244: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:54:34.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:33 smithi164 ceph-mon[88618]: pgmap v244: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:54:34.306 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:54:34.031+0000 7fe9accf07c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-03-26T02:54:34.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:54:34.062+0000 7fe9accf07c0 -1 osd.7 147 log_to_monitors true 2024-03-26T02:54:35.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:34 smithi164 ceph-mon[88618]: from='osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:54:35.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:34 smithi164 ceph-mon[88618]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:54:35.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:34 smithi164 ceph-mon[83622]: from='osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:54:35.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:34 smithi164 ceph-mon[83622]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:54:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:34 smithi172 ceph-mon[67937]: from='osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:54:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:34 smithi172 ceph-mon[67937]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:54:35.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:54:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:54:34.778+0000 7fe99e148700 -1 osd.7 147 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:54:36.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[83622]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[83622]: osdmap e150: 8 total, 7 up, 8 in 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[83622]: from='osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[83622]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[83622]: pgmap v246: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[88618]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[88618]: osdmap e150: 8 total, 7 up, 8 in 2024-03-26T02:54:36.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[88618]: from='osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[88618]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[88618]: pgmap v246: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:54:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:36.046 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:35 smithi172 ceph-mon[67937]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:54:36.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:35 smithi172 ceph-mon[67937]: osdmap e150: 8 total, 7 up, 8 in 2024-03-26T02:54:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:35 smithi172 ceph-mon[67937]: from='osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:35 smithi172 ceph-mon[67937]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi172", "root=default"]}]: dispatch 2024-03-26T02:54:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:35 smithi172 ceph-mon[67937]: pgmap v246: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 247 MiB used, 715 GiB / 715 GiB avail; 95/723 objects degraded (13.140%) 2024-03-26T02:54:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:36.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:37.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[83622]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:54:37.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[83622]: osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073] boot 2024-03-26T02:54:37.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[83622]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[83622]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[88618]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[88618]: osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073] boot 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[88618]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:36 smithi164 ceph-mon[88618]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T02:54:37.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:36 smithi172 ceph-mon[67937]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T02:54:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:36 smithi172 ceph-mon[67937]: osd.7 [v2:172.21.15.172:6824/313977073,v1:172.21.15.172:6825/313977073] boot 2024-03-26T02:54:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:36 smithi172 ceph-mon[67937]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T02:54:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:36 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:54:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:36 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:36 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:36 smithi172 ceph-mon[67937]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T02:54:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:37 smithi164 ceph-mon[83622]: pgmap v249: 161 pgs: 14 peering, 21 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-26T02:54:38.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:37 smithi164 ceph-mon[83622]: Health check update: Degraded data redundancy: 78/723 objects degraded (10.788%), 21 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:38.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:37 smithi164 ceph-mon[88618]: pgmap v249: 161 pgs: 14 peering, 21 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-26T02:54:38.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:37 smithi164 ceph-mon[88618]: Health check update: Degraded data redundancy: 78/723 objects degraded (10.788%), 21 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:37 smithi172 ceph-mon[67937]: pgmap v249: 161 pgs: 14 peering, 21 active+undersized+degraded, 34 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 78/723 objects degraded (10.788%) 2024-03-26T02:54:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:37 smithi172 ceph-mon[67937]: Health check update: Degraded data redundancy: 78/723 objects degraded (10.788%), 21 pgs degraded (PG_DEGRADED) 2024-03-26T02:54:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:38 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:38 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:38 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:39.364 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:54:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[83622]: pgmap v250: 161 pgs: 14 peering, 16 active+undersized+degraded, 27 active+undersized, 104 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/723 objects degraded (8.299%) 2024-03-26T02:54:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[88618]: pgmap v250: 161 pgs: 14 peering, 16 active+undersized+degraded, 27 active+undersized, 104 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/723 objects degraded (8.299%) 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.053 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:54:40.053 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (9m) 2m ago 31m 26.1M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:54:40.053 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (9m) 4s ago 30m 78.3M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (10m) 2m ago 30m 58.6M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (10m) 4s ago 35m 453M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (20m) 2m ago 37m 510M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (7m) 2m ago 37m 50.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (7m) 4s ago 36m 53.2M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (6m) 2m ago 36m 41.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (9m) 2m ago 31m 15.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (9m) 4s ago 31m 16.4M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (4m) 2m ago 35m 53.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (4m) 2m ago 35m 45.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (3m) 2m ago 34m 55.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:54:40.054 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (2m) 2m ago 34m 12.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:54:40.055 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (112s) 4s ago 33m 58.3M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:54:40.055 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (78s) 4s ago 33m 56.5M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:54:40.055 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (44s) 4s ago 33m 51.9M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 44e8c5cf078f 2024-03-26T02:54:40.055 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (9s) 4s ago 32m 12.7M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 66123d33bbf2 2024-03-26T02:54:40.055 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (9m) 4s ago 32m 75.2M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:54:40.055 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (30m) 2m ago 30m 79.2M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:54:40.055 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (30m) 4s ago 30m 82.5M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:54:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:39 smithi172 ceph-mon[67937]: pgmap v250: 161 pgs: 14 peering, 16 active+undersized+degraded, 27 active+undersized, 104 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 60/723 objects degraded (8.299%) 2024-03-26T02:54:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:40.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 8 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:54:40.454 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:54:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:54:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:54:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:54:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-26T02:54:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 13 2024-03-26T02:54:40.455 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:54:40.455 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons of type(s) crash,osd", 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [ 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: "osd" 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "8/8 daemons upgraded", 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading osd daemons", 2024-03-26T02:54:40.790 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:54:40.791 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='client.64540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all osd 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T02:54:41.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-26T02:54:41.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='client.64540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all osd 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T02:54:41.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-26T02:54:41.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-26T02:54:41.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T02:54:41.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/837889243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-26T02:54:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T02:54:41.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:40 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/837889243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='client.64540 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='client.64546 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all osd 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-03-26T02:54:41.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-03-26T02:54:41.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:40 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/837889243' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.044 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:54:41] "GET /metrics HTTP/1.1" 200 38306 "" "Prometheus/2.43.0" 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: pgmap v251: 161 pgs: 14 peering, 1 active+undersized+degraded, 146 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='client.64564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.046 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: pgmap v251: 161 pgs: 14 peering, 1 active+undersized+degraded, 146 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='client.64564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T02:54:42.047 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.048 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:54:42.049 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:54:42.050 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:42.051 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:54:42.052 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.053 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:42.054 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:41 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: Upgrade: Setting require_osd_release to 19 squid 2024-03-26T02:54:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='client.64552 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: pgmap v251: 161 pgs: 14 peering, 1 active+undersized+degraded, 146 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 851 B/s rd, 0 op/s; 6/723 objects degraded (0.830%) 2024-03-26T02:54:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='client.64564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:54:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-03-26T02:54:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-03-26T02:54:42.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:54:42.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:54:42.060 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:54:42.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:42.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:42.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:42.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:42.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:54:42.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:54:42.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:54:42.061 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:41 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mds 2024-03-26T02:54:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:54:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:54:43.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all nfs 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[83622]: Upgrade: Finalizing container_image settings 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[83622]: Upgrade: Complete! 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mds 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all nfs 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[88618]: Upgrade: Finalizing container_image settings 2024-03-26T02:54:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:42 smithi164 ceph-mon[88618]: Upgrade: Complete! 2024-03-26T02:54:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:42 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mds 2024-03-26T02:54:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:42 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:54:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:42 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:54:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:42 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nfs 2024-03-26T02:54:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:42 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:54:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:42 smithi172 ceph-mon[67937]: Upgrade: Finalizing container_image settings 2024-03-26T02:54:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:42 smithi172 ceph-mon[67937]: Upgrade: Complete! 2024-03-26T02:54:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:43 smithi164 ceph-mon[83622]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:54:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:43 smithi164 ceph-mon[83622]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded) 2024-03-26T02:54:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:43 smithi164 ceph-mon[83622]: Cluster is now healthy 2024-03-26T02:54:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:43 smithi164 ceph-mon[88618]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:54:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:43 smithi164 ceph-mon[88618]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded) 2024-03-26T02:54:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:43 smithi164 ceph-mon[88618]: Cluster is now healthy 2024-03-26T02:54:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:43 smithi172 ceph-mon[67937]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:54:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:43 smithi172 ceph-mon[67937]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded) 2024-03-26T02:54:44.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:43 smithi172 ceph-mon[67937]: Cluster is now healthy 2024-03-26T02:54:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:45 smithi164 ceph-mon[83622]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:54:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:45 smithi164 ceph-mon[88618]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:54:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:54:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:45 smithi172 ceph-mon[67937]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:54:46.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:47.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:46 smithi164 ceph-mon[83622]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:47.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:46 smithi164 ceph-mon[88618]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:47.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:46 smithi172 ceph-mon[67937]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:48.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:47 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:48.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:47 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:48.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:47 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:49.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:48 smithi164 ceph-mon[83622]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:49.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:48 smithi164 ceph-mon[88618]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:49.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:48 smithi172 ceph-mon[67937]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:51.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:50 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:51.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:50 smithi164 ceph-mon[83622]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:51.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:50 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:51.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:50 smithi164 ceph-mon[88618]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:51.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:50 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:54:51.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:50 smithi172 ceph-mon[67937]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:54:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:54:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:54:52] "GET /metrics HTTP/1.1" 200 38316 "" "Prometheus/2.43.0" 2024-03-26T02:54:53.123 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:52 smithi172 ceph-mon[67937]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:54:53.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:52 smithi164 ceph-mon[83622]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:54:53.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:52 smithi164 ceph-mon[88618]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:54:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:55 smithi164 ceph-mon[83622]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:54:56.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:55 smithi164 ceph-mon[88618]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:54:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:55 smithi172 ceph-mon[67937]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:54:57.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:56 smithi164 ceph-mon[83622]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:54:57.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:56 smithi164 ceph-mon[88618]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:54:57.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:56 smithi172 ceph-mon[67937]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:54:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:57 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:58.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:57 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:58.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:57 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:54:59.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:58 smithi164 ceph-mon[83622]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:54:59.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:58 smithi164 ceph-mon[88618]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:54:59.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:58 smithi172 ceph-mon[67937]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:00.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:54:59 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:00.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:54:59 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:00.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:54:59 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:01.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:00 smithi164 ceph-mon[83622]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:01.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:00 smithi164 ceph-mon[88618]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:01.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:00 smithi172 ceph-mon[67937]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:55:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:55:01] "GET /metrics HTTP/1.1" 200 38316 "" "Prometheus/2.43.0" 2024-03-26T02:55:03.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:02 smithi164 ceph-mon[83622]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:02 smithi164 ceph-mon[88618]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:02 smithi172 ceph-mon[67937]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:05.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:04 smithi164 ceph-mon[83622]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:05.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:04 smithi164 ceph-mon[88618]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:05.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:04 smithi172 ceph-mon[67937]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:07.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:06 smithi164 ceph-mon[83622]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:07.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:06 smithi164 ceph-mon[88618]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:07.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:06 smithi172 ceph-mon[67937]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:08.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:07 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:08.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:07 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:08.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:07 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:09.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:08 smithi164 ceph-mon[83622]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:09.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:08 smithi164 ceph-mon[88618]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:09.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:08 smithi172 ceph-mon[67937]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:11.154 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:10 smithi164 ceph-mon[88618]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:11.155 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:10 smithi164 ceph-mon[83622]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:11.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:10 smithi172 ceph-mon[67937]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:11.948 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:55:12.257 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:11 smithi164 ceph-mon[83622]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:12.257 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:11 smithi164 ceph-mon[88618]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:12.258 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:55:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:55:11] "GET /metrics HTTP/1.1" 200 38316 "" "Prometheus/2.43.0" 2024-03-26T02:55:12.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:11 smithi172 ceph-mon[67937]: from='client.54636 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:13.229 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:12 smithi164 ceph-mon[83622]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:13.229 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:12 smithi164 ceph-mon[88618]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:13.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:12 smithi172 ceph-mon[67937]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (9m) 2m ago 31m 26.1M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (9m) 38s ago 31m 78.3M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (10m) 2m ago 30m 58.6M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (10m) 38s ago 36m 453M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (21m) 2m ago 37m 510M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (7m) 2m ago 37m 50.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (8m) 38s ago 36m 53.2M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:55:13.939 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (7m) 2m ago 36m 41.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (10m) 2m ago 32m 15.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (10m) 38s ago 32m 16.4M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (5m) 2m ago 36m 53.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (4m) 2m ago 35m 45.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (3m) 2m ago 35m 55.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (3m) 2m ago 34m 12.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (2m) 38s ago 34m 58.3M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (112s) 38s ago 34m 56.5M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (77s) 38s ago 33m 51.9M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 44e8c5cf078f 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (43s) 38s ago 33m 12.7M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 66123d33bbf2 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (10m) 38s ago 32m 75.2M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (31m) 2m ago 31m 79.2M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:55:13.940 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (31m) 38s ago 31m 82.5M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:55:14.568 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.osd | length == 1'"'"'' 2024-03-26T02:55:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:14 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:15.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:14 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:14 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:15.952 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:15 smithi164 ceph-mon[83622]: from='client.54642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:15.952 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:15 smithi164 ceph-mon[83622]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:15.952 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:15 smithi164 ceph-mon[88618]: from='client.54642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:15.953 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:15 smithi164 ceph-mon[88618]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:16.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:15 smithi172 ceph-mon[67937]: from='client.54642 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:15 smithi172 ceph-mon[67937]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:16.767 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:55:17.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:16 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2684838787' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:17.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:16 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2684838787' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:17.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:16 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2684838787' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:17.476 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.osd | keys'"'"' | grep $sha1' 2024-03-26T02:55:18.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:17 smithi164 ceph-mon[83622]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:18.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:17 smithi164 ceph-mon[88618]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:18.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:17 smithi172 ceph-mon[67937]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:18 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:18 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:18 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:19.559 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)" 2024-03-26T02:55:19.890 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:19 smithi164 ceph-mon[83622]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:19.890 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:19 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/724928589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:19.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:19 smithi164 ceph-mon[88618]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:19.890 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:19 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/724928589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:19 smithi172 ceph-mon[67937]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:20.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:19 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/724928589' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:20.308 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade status' 2024-03-26T02:55:21.863 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:21 smithi164 ceph-mon[88618]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:21.864 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:21 smithi164 ceph-mon[83622]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:22.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:21 smithi172 ceph-mon[67937]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:55:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:55:21] "GET /metrics HTTP/1.1" 200 38316 "" "Prometheus/2.43.0" 2024-03-26T02:55:22.310 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:55:22.310 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": null, 2024-03-26T02:55:22.310 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": false, 2024-03-26T02:55:22.310 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "", 2024-03-26T02:55:22.310 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:55:22.310 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": null, 2024-03-26T02:55:22.311 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:55:22.311 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:55:22.311 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:55:23.066 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:55:24.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:23 smithi164 ceph-mon[88618]: from='client.54660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:24.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:23 smithi164 ceph-mon[88618]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:24.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:23 smithi164 ceph-mon[83622]: from='client.54660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:24.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:23 smithi164 ceph-mon[83622]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:24.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:23 smithi172 ceph-mon[67937]: from='client.54660 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:24.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:23 smithi172 ceph-mon[67937]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:25.321 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:55:25.988 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:25 smithi164 ceph-mon[83622]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:25.988 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:25 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/361241942' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:55:25.988 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:25 smithi164 ceph-mon[88618]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:25.989 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:25 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/361241942' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:55:26.011 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --services rgw.foo' 2024-03-26T02:55:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:25 smithi172 ceph-mon[67937]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:26.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:25 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/361241942' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:55:28.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:27 smithi164 ceph-mon[83622]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:28.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:27 smithi164 ceph-mon[88618]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:28.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:27 smithi172 ceph-mon[67937]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:29.161 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:28 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:29.161 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:28 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:29.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:28 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:30.133 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:29 smithi164 ceph-mon[83622]: from='client.64606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:30.133 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:29 smithi164 ceph-mon[83622]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:30.133 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:29 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:30.133 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:29 smithi164 ceph-mon[88618]: from='client.64606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:30.133 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:29 smithi164 ceph-mon[88618]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:30.133 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:29 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:30.186 INFO:teuthology.orchestra.run.smithi164.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:55:30.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:29 smithi172 ceph-mon[67937]: from='client.64606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "services": "rgw.foo", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:30.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:29 smithi172 ceph-mon[67937]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:30.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:29 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:30.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-26T02:55:31.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[83622]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:55:31.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:31.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:31.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:31.419 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[83622]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[83622]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[88618]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[88618]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:55:31.420 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:31 smithi164 ceph-mon[88618]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:31 smithi172 ceph-mon[67937]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:55:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:31 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:31 smithi172 ceph-mon[67937]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:55:31.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:31 smithi172 ceph-mon[67937]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:32.289 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:55:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:55:32] "GET /metrics HTTP/1.1" 200 38318 "" "Prometheus/2.43.0" 2024-03-26T02:55:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:32 smithi164 ceph-mon[83622]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:33.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:32 smithi164 ceph-mon[88618]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:32 smithi172 ceph-mon[67937]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:34.068 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:55:34.763 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:55:34.763 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (10m) 3m ago 32m 26.1M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:55:34.763 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (10m) 59s ago 31m 78.3M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (11m) 3m ago 31m 58.6M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (11m) 59s ago 36m 453M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (21m) 3m ago 38m 510M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (8m) 3m ago 38m 50.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (8m) 59s ago 36m 53.2M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (7m) 3m ago 37m 41.4M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (10m) 3m ago 32m 15.6M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (10m) 59s ago 32m 16.4M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (5m) 3m ago 36m 53.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (5m) 3m ago 35m 45.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (4m) 3m ago 35m 55.0M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (3m) 3m ago 35m 12.3M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (2m) 59s ago 34m 58.3M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (2m) 59s ago 34m 56.5M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:55:34.764 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (98s) 59s ago 34m 51.9M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 44e8c5cf078f 2024-03-26T02:55:34.765 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (64s) 59s ago 33m 12.7M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 66123d33bbf2 2024-03-26T02:55:34.765 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (10m) 59s ago 32m 75.2M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:55:34.765 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (31m) 3m ago 31m 79.2M - 17.2.0 e1d6a67b021e 174533750a07 2024-03-26T02:55:34.765 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (31m) 59s ago 31m 82.5M - 17.2.0 e1d6a67b021e 1aae416e5bce 2024-03-26T02:55:34.957 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.957 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mgr 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mon 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all crash 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all osd 2024-03-26T02:55:34.958 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mds 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mgr 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mon 2024-03-26T02:55:34.959 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all crash 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all osd 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mds 2024-03-26T02:55:34.960 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:34 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mgr 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mon 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all crash 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:35.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all osd 2024-03-26T02:55:35.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:35.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mds 2024-03-26T02:55:35.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:34 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.172 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:55:35.172 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:55:35.172 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:55:35.172 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:55:35.172 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:55:35.172 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 8 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 13 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:55:35.173 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:55:35.517 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:55:35.517 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:55:35.517 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:55:35.517 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading daemons in service(s) rgw.foo", 2024-03-26T02:55:35.517 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:55:35.517 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "0/2 daemons upgraded", 2024-03-26T02:55:35.518 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading rgw daemons", 2024-03-26T02:55:35.518 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:55:35.518 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:55:35.922 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-mon[67937]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-mon[67937]: from='client.64618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-mon[67937]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-mon[67937]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:35.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2535790979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:35.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:55:35.923 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:35.992 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[83622]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[83622]: from='client.64618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[83622]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[83622]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2535790979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[88618]: from='client.64612 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[88618]: from='client.64618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[88618]: from='client.64624 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[88618]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2535790979' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:35.993 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:35.994 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi164.cdxqhg", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:55:35.994 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:35 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:36.306 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=cleanup t=2024-03-26T02:55:35.921442931Z level=info msg="Completed cleanup jobs" duration=2.600865ms 2024-03-26T02:55:36.307 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 02:55:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=sqlstore.transactions t=2024-03-26T02:55:35.930294105Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T02:55:36.996 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:36 smithi164 ceph-mon[83622]: Upgrade: Updating rgw.foo.smithi164.cdxqhg (1/2) 2024-03-26T02:55:36.996 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:36 smithi164 ceph-mon[83622]: Deploying daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:55:36.996 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:36 smithi164 ceph-mon[83622]: from='client.54702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:36.996 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:36 smithi164 ceph-mon[88618]: Upgrade: Updating rgw.foo.smithi164.cdxqhg (1/2) 2024-03-26T02:55:36.996 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:36 smithi164 ceph-mon[88618]: Deploying daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:55:36.996 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:36 smithi164 ceph-mon[88618]: from='client.54702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:37.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:36 smithi172 ceph-mon[67937]: Upgrade: Updating rgw.foo.smithi164.cdxqhg (1/2) 2024-03-26T02:55:37.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:36 smithi172 ceph-mon[67937]: Deploying daemon rgw.foo.smithi164.cdxqhg on smithi164 2024-03-26T02:55:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:36 smithi172 ceph-mon[67937]: from='client.54702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:55:37.982 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:37 smithi164 ceph-mon[83622]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:37.983 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:37 smithi164 ceph-mon[88618]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:38.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:37 smithi172 ceph-mon[67937]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:55:38.915 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:38 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:38.915 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:38 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:38 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:39 smithi164 ceph-mon[83622]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:39 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:39 smithi164 ceph-mon[88618]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:40.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:39 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:39 smithi172 ceph-mon[67937]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:55:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:39 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:41.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:40 smithi164 ceph-mon[83622]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 85 B/s wr, 59 op/s 2024-03-26T02:55:41.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:40 smithi164 ceph-mon[88618]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 85 B/s wr, 59 op/s 2024-03-26T02:55:41.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:40 smithi172 ceph-mon[67937]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 85 B/s wr, 59 op/s 2024-03-26T02:55:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:55:42] "GET /metrics HTTP/1.1" 200 38318 "" "Prometheus/2.43.0" 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[83622]: Upgrade: Updating rgw.foo.smithi172.gbbkmx (2/2) 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[83622]: Deploying daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[88618]: Upgrade: Updating rgw.foo.smithi172.gbbkmx (2/2) 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:42.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:42 smithi164 ceph-mon[88618]: Deploying daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:55:42.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:42 smithi172 ceph-mon[67937]: Upgrade: Updating rgw.foo.smithi172.gbbkmx (2/2) 2024-03-26T02:55:42.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:42.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi172.gbbkmx", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:55:42.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:42.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:42 smithi172 ceph-mon[67937]: Deploying daemon rgw.foo.smithi172.gbbkmx on smithi172 2024-03-26T02:55:43.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:43 smithi164 ceph-mon[83622]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 85 B/s wr, 103 op/s 2024-03-26T02:55:43.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:43 smithi164 ceph-mon[88618]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 85 B/s wr, 103 op/s 2024-03-26T02:55:43.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:43 smithi172 ceph-mon[67937]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 85 B/s wr, 103 op/s 2024-03-26T02:55:45.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:45.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:44 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:45.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:55:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:45 smithi164 ceph-mon[83622]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T02:55:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:46.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:45 smithi164 ceph-mon[88618]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T02:55:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:45 smithi172 ceph-mon[67937]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-03-26T02:55:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:46.956 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:46 smithi164 ceph-mon[83622]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 170 B/s wr, 175 op/s 2024-03-26T02:55:46.957 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:46 smithi164 ceph-mon[88618]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 170 B/s wr, 175 op/s 2024-03-26T02:55:47.277 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:46 smithi172 ceph-mon[67937]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 170 B/s wr, 175 op/s 2024-03-26T02:55:48.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:47 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:48.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:47 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:48.202 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:47 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[83622]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 170 B/s wr, 198 op/s 2024-03-26T02:55:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:49 smithi164 ceph-mon[88618]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 170 B/s wr, 198 op/s 2024-03-26T02:55:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:49 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:49 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:49 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:49 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:50.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:49 smithi172 ceph-mon[67937]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 170 B/s wr, 198 op/s 2024-03-26T02:55:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:50 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:50 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:50 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:50 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:50 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:50 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:50 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:50 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:50 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:50 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:50 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:50 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:51.951 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:51 smithi164 ceph-mon[83622]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T02:55:51.951 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:51 smithi164 ceph-mon[88618]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T02:55:52.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:51 smithi172 ceph-mon[67937]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-03-26T02:55:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:55:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:55:51] "GET /metrics HTTP/1.1" 200 38316 "" "Prometheus/2.43.0" 2024-03-26T02:55:53.038 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:52 smithi164 ceph-mon[83622]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 85 B/s wr, 189 op/s 2024-03-26T02:55:53.039 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:52 smithi164 ceph-mon[88618]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 85 B/s wr, 189 op/s 2024-03-26T02:55:53.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:52 smithi172 ceph-mon[67937]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 85 B/s wr, 189 op/s 2024-03-26T02:55:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all rgw 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi164.cdxqhg"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi164.cdxqhg"}]': finished 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.gbbkmx"}]: dispatch 2024-03-26T02:55:54.308 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.gbbkmx"}]': finished 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nfs 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.309 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: Upgrade: Finalizing container_image settings 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:55:54.310 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:55:54.311 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: Upgrade: Complete! 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: Checking dashboard <-> RGW credentials 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.312 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.313 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.313 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:54 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all rgw 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi164.cdxqhg"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi164.cdxqhg"}]': finished 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.gbbkmx"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.gbbkmx"}]': finished 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:55:54.546 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all nfs 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.547 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all rgw 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi164.cdxqhg"}]: dispatch 2024-03-26T02:55:54.548 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi164.cdxqhg"}]': finished 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.gbbkmx"}]: dispatch 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi172.gbbkmx"}]': finished 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all nfs 2024-03-26T02:55:54.549 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: Upgrade: Finalizing container_image settings 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.550 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.551 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all nvmeof 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: Upgrade: Finalizing container_image settings 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.552 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.553 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: Upgrade: Complete! 2024-03-26T02:55:54.554 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: Checking dashboard <-> RGW credentials 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-03-26T02:55:54.555 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: Upgrade: Complete! 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.556 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.557 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.557 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: Checking dashboard <-> RGW credentials 2024-03-26T02:55:54.557 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:54.557 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:55:54.557 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:55:54.557 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:55:54.557 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:54 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:55.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:55 smithi164 ceph-mon[83622]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 146 op/s 2024-03-26T02:55:55.544 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:55 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:55.544 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:55 smithi164 ceph-mon[88618]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 146 op/s 2024-03-26T02:55:55.545 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:55 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:55.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:55 smithi172 ceph-mon[67937]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s rd, 85 B/s wr, 146 op/s 2024-03-26T02:55:55.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:55 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:55:57.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:56 smithi164 ceph-mon[83622]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 85 B/s wr, 126 op/s 2024-03-26T02:55:57.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:56 smithi164 ceph-mon[88618]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 85 B/s wr, 126 op/s 2024-03-26T02:55:57.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:56 smithi172 ceph-mon[67937]: pgmap v290: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 85 B/s wr, 126 op/s 2024-03-26T02:55:58.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:57 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:58.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:57 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:58.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:57 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:55:59.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:58 smithi164 ceph-mon[83622]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-26T02:55:59.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:58 smithi164 ceph-mon[88618]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-26T02:55:59.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:58 smithi172 ceph-mon[67937]: pgmap v291: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-26T02:56:00.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:55:59 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:00.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:55:59 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:00.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:55:59 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:01.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:00 smithi164 ceph-mon[83622]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-26T02:56:01.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:00 smithi164 ceph-mon[88618]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-26T02:56:01.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:00 smithi172 ceph-mon[67937]: pgmap v292: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 34 KiB/s rd, 0 B/s wr, 52 op/s 2024-03-26T02:56:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:56:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:56:02] "GET /metrics HTTP/1.1" 200 38350 "" "Prometheus/2.43.0" 2024-03-26T02:56:03.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:02 smithi164 ceph-mon[83622]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-03-26T02:56:03.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:02 smithi164 ceph-mon[88618]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-03-26T02:56:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:02 smithi172 ceph-mon[67937]: pgmap v293: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 3.1 KiB/s rd, 3 op/s 2024-03-26T02:56:05.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:04 smithi164 ceph-mon[83622]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T02:56:05.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:04 smithi164 ceph-mon[88618]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T02:56:05.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:04 smithi172 ceph-mon[67937]: pgmap v294: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T02:56:06.534 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:56:07.162 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:06 smithi164 ceph-mon[83622]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:07.162 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:06 smithi164 ceph-mon[83622]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T02:56:07.162 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:06 smithi164 ceph-mon[88618]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:07.163 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:06 smithi164 ceph-mon[88618]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T02:56:07.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:06 smithi172 ceph-mon[67937]: from='client.54786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:07.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:06 smithi172 ceph-mon[67937]: pgmap v295: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2 op/s 2024-03-26T02:56:08.154 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:07 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:08.154 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:07 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:08.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:07 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:08.680 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:56:08.680 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (10m) 20s ago 32m 26.0M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:56:08.680 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (10m) 19s ago 32m 80.9M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:56:08.680 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (11m) 20s ago 31m 58.7M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:56:08.680 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (11m) 19s ago 37m 454M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:56:08.680 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (21m) 20s ago 38m 515M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (8m) 20s ago 38m 58.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (9m) 19s ago 37m 56.2M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (8m) 20s ago 37m 48.5M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (11m) 20s ago 33m 16.0M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (11m) 19s ago 33m 16.1M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (6m) 20s ago 36m 62.4M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (5m) 20s ago 36m 56.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (4m) 20s ago 36m 61.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (3m) 20s ago 35m 60.6M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (3m) 19s ago 35m 64.0M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (2m) 19s ago 34m 60.6M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (2m) 19s ago 34m 55.7M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 44e8c5cf078f 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (98s) 19s ago 34m 47.8M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 66123d33bbf2 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (11m) 19s ago 33m 79.3M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:56:08.681 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (29s) 20s ago 31m 77.6M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f7f6fade545f 2024-03-26T02:56:08.682 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (24s) 19s ago 31m 75.3M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fc0285a1be3 2024-03-26T02:56:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:08 smithi164 ceph-mon[83622]: from='client.64726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:08 smithi164 ceph-mon[83622]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:56:09.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:08 smithi164 ceph-mon[88618]: from='client.64726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:09.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:08 smithi164 ceph-mon[88618]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:56:09.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:08 smithi172 ceph-mon[67937]: from='client.64726 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:09.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:08 smithi172 ceph-mon[67937]: pgmap v296: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:56:09.442 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.rgw | length == 1'"'"'' 2024-03-26T02:56:11.141 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:10 smithi164 ceph-mon[83622]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:11.142 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:10 smithi164 ceph-mon[88618]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:11.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:10 smithi172 ceph-mon[67937]: pgmap v297: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:11.746 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:56:12.000 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:11 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/4248360138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:12.000 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:11 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/4248360138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:56:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:56:11] "GET /metrics HTTP/1.1" 200 38350 "" "Prometheus/2.43.0" 2024-03-26T02:56:12.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:11 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/4248360138' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:12.468 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.rgw | keys'"'"' | grep $sha1' 2024-03-26T02:56:13.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:12 smithi164 ceph-mon[83622]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:13.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:12 smithi164 ceph-mon[88618]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:13.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:12 smithi172 ceph-mon[67937]: pgmap v298: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:14.727 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)" 2024-03-26T02:56:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:14 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2480253173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:14 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:14 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2480253173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:14 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:14 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2480253173' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:14 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:15.415 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade status' 2024-03-26T02:56:16.012 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:15 smithi164 ceph-mon[83622]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:16.012 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:15 smithi164 ceph-mon[88618]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:16.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:15 smithi172 ceph-mon[67937]: pgmap v299: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:17.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:16 smithi164 ceph-mon[83622]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:17.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:16 smithi164 ceph-mon[88618]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:17.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:16 smithi172 ceph-mon[67937]: pgmap v300: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:17.616 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:56:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": null, 2024-03-26T02:56:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": false, 2024-03-26T02:56:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "", 2024-03-26T02:56:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [], 2024-03-26T02:56:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": null, 2024-03-26T02:56:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "", 2024-03-26T02:56:17.617 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:56:17.617 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:56:18.206 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:17 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:18.206 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:17 smithi164 ceph-mon[83622]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:18.206 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:17 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:18.206 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:17 smithi164 ceph-mon[88618]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:18.228 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:56:18.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:17 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:18.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:17 smithi172 ceph-mon[67937]: from='client.64744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:18 smithi164 ceph-mon[88618]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:19.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:18 smithi164 ceph-mon[83622]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:19.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:18 smithi172 ceph-mon[67937]: pgmap v301: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:20.338 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:56:20.687 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:20 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/1101221051' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:56:20.687 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:20 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/1101221051' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:56:20.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:20 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/1101221051' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:56:21.011 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-03-26T02:56:21.574 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:21 smithi164 ceph-mon[83622]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:21.574 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:21 smithi164 ceph-mon[88618]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:21.806 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:21 smithi172 ceph-mon[67937]: pgmap v302: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:22.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:56:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:56:22] "GET /metrics HTTP/1.1" 200 38348 "" "Prometheus/2.43.0" 2024-03-26T02:56:23.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:22 smithi164 ceph-mon[83622]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:23.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:22 smithi164 ceph-mon[88618]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:23.127 INFO:teuthology.orchestra.run.smithi164.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:56:23.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:22 smithi172 ceph-mon[67937]: pgmap v303: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:23.942 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T02:56:23.954 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi164.front.sepia.ceph.com 2024-03-26T02:56:23.955 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-03-26T02:56:24.268 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[83622]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:24.268 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[83622]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:56:24.268 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:24.268 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[83622]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[88618]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[88618]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:24.269 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:24.270 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:24.270 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:24.270 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:24 smithi164 ceph-mon[88618]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:56:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:24 smithi172 ceph-mon[67937]: from='client.64756 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:24 smithi172 ceph-mon[67937]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:56:24.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:24 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:24.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:24 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:24.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:24 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:24.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:24 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:24.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:24 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:24.558 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:24 smithi172 ceph-mon[67937]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T02:56:25.482 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:25 smithi164 ceph-mon[83622]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:25.482 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:25 smithi164 ceph-mon[88618]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:25.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:25 smithi172 ceph-mon[67937]: pgmap v304: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:27.252 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mgr 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mon 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all crash 2024-03-26T02:56:27.795 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all osd 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all mds 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all rgw 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:56:27.796 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[83622]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mgr 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mon 2024-03-26T02:56:27.797 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all crash 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all osd 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all mds 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all rgw 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.798 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:56:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:56:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:56:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:27.799 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:27 smithi164 ceph-mon[88618]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (11m) 39s ago 33m 26.0M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (10m) 39s ago 32m 80.9M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (12m) 39s ago 32m 58.7M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (12m) 39s ago 37m 454M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (22m) 39s ago 39m 515M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (8m) 39s ago 39m 58.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (9m) 39s ago 37m 56.2M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (8m) 39s ago 37m 48.5M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:56:27.931 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (11m) 39s ago 33m 16.0M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (11m) 39s ago 33m 16.1M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (6m) 39s ago 37m 62.4M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (6m) 39s ago 36m 56.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (5m) 39s ago 36m 61.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (4m) 39s ago 36m 60.6M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (3m) 39s ago 35m 64.0M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (3m) 39s ago 35m 60.6M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (2m) 39s ago 34m 55.7M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 44e8c5cf078f 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (117s) 39s ago 34m 47.8M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 66123d33bbf2 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (11m) 39s ago 33m 79.3M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (48s) 39s ago 32m 77.6M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f7f6fade545f 2024-03-26T02:56:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (43s) 39s ago 32m 75.3M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fc0285a1be3 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Target is version 19.0.0-2475-ge42b8ea3 (squid) 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, digests ['quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717'] 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mgr 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mon 2024-03-26T02:56:28.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all crash 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all osd 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all mds 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all rgw 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.058 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all rbd-mirror 2024-03-26T02:56:28.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all cephfs-mirror 2024-03-26T02:56:28.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: Upgrade: Setting container_image for all ceph-exporter 2024-03-26T02:56:28.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.059 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:27 smithi172 ceph-mon[67937]: pgmap v305: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 8 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:56:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:56:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:56:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:56:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 15 2024-03-26T02:56:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:56:28.340 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:56:28.695 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:56:28.695 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [ 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw", 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "osd", 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr", 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "mon" 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "15/21 daemons upgraded", 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Currently upgrading iscsi daemons", 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": false 2024-03-26T02:56:28.696 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:56:28.873 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[83622]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[83622]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/3358527651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[88618]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[88618]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:28.874 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:28.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:56:28.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:28.875 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:28 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/3358527651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:28 smithi172 ceph-mon[67937]: from='client.64762 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:28 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:28 smithi172 ceph-mon[67937]: from='client.64768 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:28 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:28 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi164.bueptz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:56:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:28 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:29.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:28 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/3358527651' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:56:29.114 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_OK 2024-03-26T02:56:29.876 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[83622]: from='client.64774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[83622]: Upgrade: Updating iscsi.foo.smithi164.bueptz 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[83622]: Deploying daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[83622]: from='client.64786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[83622]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2080500888' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[88618]: from='client.64774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[88618]: Upgrade: Updating iscsi.foo.smithi164.bueptz 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[88618]: Deploying daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[88618]: from='client.64786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[88618]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:29.877 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:29 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2080500888' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:56:30.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:29 smithi172 ceph-mon[67937]: from='client.64774 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:29 smithi172 ceph-mon[67937]: Upgrade: Updating iscsi.foo.smithi164.bueptz 2024-03-26T02:56:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:29 smithi172 ceph-mon[67937]: Deploying daemon iscsi.foo.smithi164.bueptz on smithi164 2024-03-26T02:56:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:29 smithi172 ceph-mon[67937]: from='client.64786 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:56:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:29 smithi172 ceph-mon[67937]: pgmap v306: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:30.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:29 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2080500888' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:56:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:30 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:30 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:30 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:32.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:31 smithi164 ceph-mon[83622]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:32.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:31 smithi164 ceph-mon[88618]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:32.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:31 smithi172 ceph-mon[67937]: pgmap v307: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:56:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:56:32] "GET /metrics HTTP/1.1" 200 38348 "" "Prometheus/2.43.0" 2024-03-26T02:56:33.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:32 smithi164 ceph-mon[83622]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:33.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:32 smithi164 ceph-mon[88618]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:32 smithi172 ceph-mon[67937]: pgmap v308: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:35.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:34 smithi164 ceph-mon[83622]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:35.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:34 smithi164 ceph-mon[88618]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:35.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:34 smithi172 ceph-mon[67937]: pgmap v309: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:37.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:36 smithi164 ceph-mon[83622]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:37.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:36 smithi164 ceph-mon[88618]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:37.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:36 smithi172 ceph-mon[67937]: pgmap v310: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:38.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:37 smithi164 ceph-mon[83622]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:38.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:37 smithi164 ceph-mon[88618]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:38.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:37 smithi172 ceph-mon[67937]: from='client.15465 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:56:39.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:38 smithi164 ceph-mon[83622]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:39.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:38 smithi164 ceph-mon[88618]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:39.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:38 smithi172 ceph-mon[67937]: pgmap v311: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:40 smithi164 ceph-mon[83622]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:41.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:40 smithi164 ceph-mon[88618]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:41.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:40 smithi172 ceph-mon[67937]: pgmap v312: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:56:42.000 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:56:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:56:41.684+0000 7f6daf7fa700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. 2024-03-26T02:56:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:56:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:56:41] "GET /metrics HTTP/1.1" 200 38348 "" "Prometheus/2.43.0" 2024-03-26T02:56:42.756 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[88618]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[88618]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[83622]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[83622]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:42.757 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:42 smithi172 ceph-mon[67937]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. 2024-03-26T02:56:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:56:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:42 smithi172 ceph-mon[67937]: pgmap v313: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 0 op/s 2024-03-26T02:56:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:56:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:56:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:44.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:43 smithi164 ceph-mon[83622]: Health check failed: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-26T02:56:44.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:43 smithi164 ceph-mon[88618]: Health check failed: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-26T02:56:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:43 smithi172 ceph-mon[67937]: Health check failed: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-03-26T02:56:45.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:44 smithi164 ceph-mon[83622]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-03-26T02:56:45.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:44 smithi164 ceph-mon[88618]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-03-26T02:56:45.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:44 smithi172 ceph-mon[67937]: pgmap v314: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-03-26T02:56:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:46.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:45 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:46.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:45 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:56:46.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:45 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:56:47.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:46 smithi164 ceph-mon[83622]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-03-26T02:56:47.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:46 smithi164 ceph-mon[88618]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-03-26T02:56:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:46 smithi172 ceph-mon[67937]: pgmap v315: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 846 B/s rd, 0 op/s 2024-03-26T02:56:49.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:48 smithi164 ceph-mon[83622]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:49.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:48 smithi164 ceph-mon[88618]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:49.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:48 smithi172 ceph-mon[67937]: pgmap v316: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:51.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:50 smithi164 ceph-mon[83622]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:51.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:50 smithi164 ceph-mon[88618]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:51.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:50 smithi172 ceph-mon[67937]: pgmap v317: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:52.070 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:56:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:56:51] "GET /metrics HTTP/1.1" 200 38426 "" "Prometheus/2.43.0" 2024-03-26T02:56:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:52 smithi164 ceph-mon[83622]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:53.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:52 smithi164 ceph-mon[88618]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:53.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:52 smithi172 ceph-mon[67937]: pgmap v318: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 376 B/s rd, 0 op/s 2024-03-26T02:56:54.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:53 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/613700086' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:56:54.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:53 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2297872038' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/321068523"}]: dispatch 2024-03-26T02:56:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:53 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/613700086' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:56:54.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:53 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2297872038' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/321068523"}]: dispatch 2024-03-26T02:56:54.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:53 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/613700086' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:56:54.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:53 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2297872038' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/321068523"}]: dispatch 2024-03-26T02:56:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:54 smithi164 ceph-mon[83622]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T02:56:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:54 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2297872038' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/321068523"}]': finished 2024-03-26T02:56:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:54 smithi164 ceph-mon[83622]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T02:56:55.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:54 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/745828686' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/321068523"}]: dispatch 2024-03-26T02:56:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:54 smithi164 ceph-mon[88618]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T02:56:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:54 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2297872038' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/321068523"}]': finished 2024-03-26T02:56:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:54 smithi164 ceph-mon[88618]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T02:56:55.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:54 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/745828686' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/321068523"}]: dispatch 2024-03-26T02:56:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:54 smithi172 ceph-mon[67937]: pgmap v319: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-03-26T02:56:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:54 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2297872038' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6801/321068523"}]': finished 2024-03-26T02:56:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:54 smithi172 ceph-mon[67937]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T02:56:55.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:54 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/745828686' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/321068523"}]: dispatch 2024-03-26T02:56:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:55 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/745828686' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/321068523"}]': finished 2024-03-26T02:56:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:55 smithi164 ceph-mon[83622]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T02:56:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:55 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2007433572' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]: dispatch 2024-03-26T02:56:56.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:55 smithi164 ceph-mon[83622]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]: dispatch 2024-03-26T02:56:56.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:55 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/745828686' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/321068523"}]': finished 2024-03-26T02:56:56.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:55 smithi164 ceph-mon[88618]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T02:56:56.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:55 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2007433572' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]: dispatch 2024-03-26T02:56:56.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:55 smithi164 ceph-mon[88618]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]: dispatch 2024-03-26T02:56:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:55 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/745828686' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:6800/321068523"}]': finished 2024-03-26T02:56:56.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:55 smithi172 ceph-mon[67937]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T02:56:56.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:55 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2007433572' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]: dispatch 2024-03-26T02:56:56.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:55 smithi172 ceph-mon[67937]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]: dispatch 2024-03-26T02:56:57.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[88618]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:56:57.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[88618]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]': finished 2024-03-26T02:56:57.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[88618]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T02:56:57.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/3267133523' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]: dispatch 2024-03-26T02:56:57.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[88618]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]: dispatch 2024-03-26T02:56:57.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[83622]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:56:57.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[83622]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]': finished 2024-03-26T02:56:57.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[83622]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T02:56:57.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/3267133523' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]: dispatch 2024-03-26T02:56:57.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:56 smithi164 ceph-mon[83622]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]: dispatch 2024-03-26T02:56:57.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:56 smithi172 ceph-mon[67937]: pgmap v322: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:56:57.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:56 smithi172 ceph-mon[67937]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/279274714"}]': finished 2024-03-26T02:56:57.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:56 smithi172 ceph-mon[67937]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T02:56:57.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:56 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/3267133523' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]: dispatch 2024-03-26T02:56:57.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:56 smithi172 ceph-mon[67937]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]: dispatch 2024-03-26T02:56:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:57 smithi164 ceph-mon[83622]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]': finished 2024-03-26T02:56:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:57 smithi164 ceph-mon[83622]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T02:56:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:57 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/4082895484' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]: dispatch 2024-03-26T02:56:58.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:57 smithi164 ceph-mon[83622]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]: dispatch 2024-03-26T02:56:58.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:57 smithi164 ceph-mon[88618]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]': finished 2024-03-26T02:56:58.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:57 smithi164 ceph-mon[88618]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T02:56:58.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:57 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/4082895484' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]: dispatch 2024-03-26T02:56:58.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:57 smithi164 ceph-mon[88618]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]: dispatch 2024-03-26T02:56:58.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:57 smithi172 ceph-mon[67937]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/498474974"}]': finished 2024-03-26T02:56:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:57 smithi172 ceph-mon[67937]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T02:56:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:57 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/4082895484' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]: dispatch 2024-03-26T02:56:58.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:57 smithi172 ceph-mon[67937]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]: dispatch 2024-03-26T02:56:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[83622]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[83622]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]': finished 2024-03-26T02:56:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[83622]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T02:56:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/1513633923' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]: dispatch 2024-03-26T02:56:59.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[83622]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]: dispatch 2024-03-26T02:56:59.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[88618]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:59.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[88618]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]': finished 2024-03-26T02:56:59.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[88618]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T02:56:59.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/1513633923' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]: dispatch 2024-03-26T02:56:59.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:58 smithi164 ceph-mon[88618]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]: dispatch 2024-03-26T02:56:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:58 smithi172 ceph-mon[67937]: pgmap v325: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:56:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:58 smithi172 ceph-mon[67937]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/2606717709"}]': finished 2024-03-26T02:56:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:58 smithi172 ceph-mon[67937]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T02:56:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:58 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/1513633923' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]: dispatch 2024-03-26T02:56:59.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:58 smithi172 ceph-mon[67937]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]: dispatch 2024-03-26T02:56:59.478 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:56:59.818 INFO:teuthology.orchestra.run.smithi164.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed." 2024-03-26T02:57:00.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:59 smithi172 ceph-mon[67937]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]': finished 2024-03-26T02:57:00.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:59 smithi172 ceph-mon[67937]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T02:57:00.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:56:59 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:00.133 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:59 smithi164 ceph-mon[83622]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]': finished 2024-03-26T02:57:00.133 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:59 smithi164 ceph-mon[83622]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T02:57:00.133 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:56:59 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:00.134 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:59 smithi164 ceph-mon[88618]: from='client.? ' entity='client.iscsi.foo.smithi164.bueptz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.164:0/1673899793"}]': finished 2024-03-26T02:57:00.134 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:59 smithi164 ceph-mon[88618]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T02:57:00.134 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:56:59 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:00.592 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:57:00.905 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:00 smithi164 ceph-mon[88618]: from='client.64858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:00.905 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:00 smithi164 ceph-mon[88618]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:00.905 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:00 smithi164 ceph-mon[88618]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:00.905 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:00 smithi164 ceph-mon[83622]: from='client.64858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:00.905 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:00 smithi164 ceph-mon[83622]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:00.906 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:00 smithi164 ceph-mon[83622]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:00 smithi172 ceph-mon[67937]: from='client.64858 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:00 smithi172 ceph-mon[67937]: pgmap v328: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:01.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:00 smithi172 ceph-mon[67937]: from='client.64864 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:02.220 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:57:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:57:01] "GET /metrics HTTP/1.1" 200 38426 "" "Prometheus/2.43.0" 2024-03-26T02:57:02.657 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (11m) 73s ago 33m 26.0M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (11m) 73s ago 33m 80.9M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (12m) 73s ago 32m 58.7M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (12m) 73s ago 38m 454M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (22m) 73s ago 39m 515M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (9m) 73s ago 39m 58.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (10m) 73s ago 38m 56.2M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (9m) 73s ago 38m 48.5M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (12m) 73s ago 34m 16.0M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (12m) 73s ago 34m 16.1M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (7m) 73s ago 37m 62.4M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:57:02.658 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (6m) 73s ago 37m 56.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (5m) 73s ago 37m 61.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (4m) 73s ago 36m 60.6M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (4m) 73s ago 36m 64.0M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (3m) 73s ago 35m 60.6M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (3m) 73s ago 35m 55.7M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 44e8c5cf078f 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (2m) 73s ago 35m 47.8M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 66123d33bbf2 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (11m) 73s ago 34m 79.3M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (83s) 73s ago 32m 77.6M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f7f6fade545f 2024-03-26T02:57:02.659 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (78s) 73s ago 32m 75.3M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fc0285a1be3 2024-03-26T02:57:03.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:02 smithi172 ceph-mon[67937]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 1 op/s 2024-03-26T02:57:03.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:02 smithi164 ceph-mon[83622]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 1 op/s 2024-03-26T02:57:03.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:02 smithi164 ceph-mon[88618]: pgmap v329: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 863 B/s rd, 1 op/s 2024-03-26T02:57:03.400 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions' 2024-03-26T02:57:03.919 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:03 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:03.919 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:03 smithi164 ceph-mon[88618]: from='client.64870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:03.920 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:03 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:03.920 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:03 smithi164 ceph-mon[83622]: from='client.64870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:04.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:03 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:04.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:03 smithi172 ceph-mon[67937]: from='client.64870 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:57:04.932 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:04 smithi164 ceph-mon[83622]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:57:04.932 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:04 smithi164 ceph-mon[88618]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:57:05.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:04 smithi172 ceph-mon[67937]: pgmap v330: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:57:05.631 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:57:05.631 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:57:05.631 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:57:05.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 8 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 15 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:57:05.632 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:57:05.923 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:05 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/521493680' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:57:05.923 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:05 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/521493680' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:57:06.273 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-03-26T02:57:06.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:05 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/521493680' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:57:06.911 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:06 smithi164 ceph-mon[83622]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:57:06.911 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:06 smithi164 ceph-mon[88618]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:57:07.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:06 smithi172 ceph-mon[67937]: pgmap v331: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:57:08.051 INFO:teuthology.orchestra.run.smithi164.stdout:wait for servicemap items w/ changing names to refresh 2024-03-26T02:57:08.734 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'sleep 60' 2024-03-26T02:57:09.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:08 smithi164 ceph-mon[88618]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:57:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:08 smithi164 ceph-mon[83622]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:57:09.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:08 smithi172 ceph-mon[67937]: pgmap v332: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:57:11.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:10 smithi164 ceph-mon[83622]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T02:57:11.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:10 smithi164 ceph-mon[88618]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T02:57:11.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:10 smithi172 ceph-mon[67937]: pgmap v333: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T02:57:12.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:57:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:57:12] "GET /metrics HTTP/1.1" 200 38426 "" "Prometheus/2.43.0" 2024-03-26T02:57:13.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:12 smithi164 ceph-mon[83622]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:13.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:12 smithi164 ceph-mon[88618]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:13.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:12 smithi172 ceph-mon[67937]: pgmap v334: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:14.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:13 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:14.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:13 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:14.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:13 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:15.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:14 smithi164 ceph-mon[88618]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:15.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:14 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:15.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:14 smithi164 ceph-mon[83622]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:15.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:14 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:15.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:14 smithi172 ceph-mon[67937]: pgmap v335: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:15.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:14 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:17.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:16 smithi164 ceph-mon[83622]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:17.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:16 smithi164 ceph-mon[88618]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:17.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:16 smithi172 ceph-mon[67937]: pgmap v336: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:19.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:18 smithi164 ceph-mon[83622]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:19.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:18 smithi164 ceph-mon[88618]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:19.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:18 smithi172 ceph-mon[67937]: pgmap v337: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:21.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:20 smithi164 ceph-mon[83622]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:21.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:20 smithi164 ceph-mon[88618]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:21.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:20 smithi172 ceph-mon[67937]: pgmap v338: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:22.253 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:57:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:57:22] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.43.0" 2024-03-26T02:57:23.109 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:22 smithi172 ceph-mon[67937]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:23.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:22 smithi164 ceph-mon[83622]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:23.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:22 smithi164 ceph-mon[88618]: pgmap v339: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:24.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:23 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:24.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:23 smithi164 ceph-mon[83622]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:23 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:24.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:23 smithi164 ceph-mon[88618]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:24.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:23 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:24.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:23 smithi172 ceph-mon[67937]: pgmap v340: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:27.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:26 smithi164 ceph-mon[83622]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:27.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:26 smithi164 ceph-mon[88618]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:27.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:26 smithi172 ceph-mon[67937]: pgmap v341: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:29.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:28 smithi164 ceph-mon[83622]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:29.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:28 smithi164 ceph-mon[88618]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:29.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:28 smithi172 ceph-mon[67937]: pgmap v342: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:30.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:29 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:30.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:29 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:30.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:29 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:30 smithi164 ceph-mon[83622]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:57:31.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:30 smithi164 ceph-mon[88618]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:57:31.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:30 smithi172 ceph-mon[67937]: pgmap v343: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:57:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:57:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:57:32] "GET /metrics HTTP/1.1" 200 38424 "" "Prometheus/2.43.0" 2024-03-26T02:57:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:32 smithi164 ceph-mon[83622]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:33.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:32 smithi164 ceph-mon[88618]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:33.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:32 smithi172 ceph-mon[67937]: pgmap v344: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:34.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:33 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:33 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:34.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:33 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:35.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:34 smithi164 ceph-mon[83622]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:35.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:34 smithi164 ceph-mon[88618]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:35.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:34 smithi172 ceph-mon[67937]: pgmap v345: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:37.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:36 smithi164 ceph-mon[83622]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:37.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:36 smithi164 ceph-mon[88618]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:37.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:36 smithi172 ceph-mon[67937]: pgmap v346: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:39.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:38 smithi164 ceph-mon[83622]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:39.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:38 smithi164 ceph-mon[88618]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:39.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:38 smithi172 ceph-mon[67937]: pgmap v347: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:41.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:40 smithi164 ceph-mon[83622]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:41.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:40 smithi164 ceph-mon[88618]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:41.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:40 smithi172 ceph-mon[67937]: pgmap v348: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:42.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:57:41] "GET /metrics HTTP/1.1" 200 38424 "" "Prometheus/2.43.0" 2024-03-26T02:57:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[88618]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:43.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:43.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:57:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[83622]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:43.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:42 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:57:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:42 smithi172 ceph-mon[67937]: pgmap v349: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:57:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:57:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:57:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:42 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' 2024-03-26T02:57:44.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:43 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:44.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:43 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:44.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:43 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:45.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:44 smithi172 ceph-mon[67937]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:45.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:44 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:45.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:44 smithi164 ceph-mon[88618]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:45.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:44 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:44 smithi164 ceph-mon[83622]: pgmap v350: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:45.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:44 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:57:47.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:46 smithi172 ceph-mon[67937]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:47.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:46 smithi164 ceph-mon[88618]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:47.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:46 smithi164 ceph-mon[83622]: pgmap v351: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:49.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:48 smithi164 ceph-mon[88618]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:49.295 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:48 smithi164 ceph-mon[83622]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:49.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:48 smithi172 ceph-mon[67937]: pgmap v352: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:51.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:50 smithi164 ceph-mon[83622]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:51.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:50 smithi164 ceph-mon[88618]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:51.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:50 smithi172 ceph-mon[67937]: pgmap v353: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:52.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:57:52 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:57:52] "GET /metrics HTTP/1.1" 200 38424 "" "Prometheus/2.43.0" 2024-03-26T02:57:53.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:52 smithi164 ceph-mon[83622]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:53.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:52 smithi164 ceph-mon[88618]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:53.108 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:52 smithi172 ceph-mon[67937]: pgmap v354: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:54.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:53 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:54.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:53 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:54.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:53 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:57:55.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:54 smithi164 ceph-mon[83622]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:55.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:54 smithi164 ceph-mon[88618]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:55.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:54 smithi172 ceph-mon[67937]: pgmap v355: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:57.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:56 smithi164 ceph-mon[83622]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:57.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:56 smithi164 ceph-mon[88618]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:57.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:56 smithi172 ceph-mon[67937]: pgmap v356: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:57:59.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:58 smithi164 ceph-mon[83622]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:59.295 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:58 smithi164 ceph-mon[88618]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:57:59.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:58 smithi172 ceph-mon[67937]: pgmap v357: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:00.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:57:59 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:00.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:57:59 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:00.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:57:59 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:01.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:00 smithi164 ceph-mon[83622]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:01.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:00 smithi164 ceph-mon[88618]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:01.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:00 smithi172 ceph-mon[67937]: pgmap v358: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:02.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:58:02] "GET /metrics HTTP/1.1" 200 38425 "" "Prometheus/2.43.0" 2024-03-26T02:58:03.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:02 smithi164 ceph-mon[83622]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:03.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:02 smithi164 ceph-mon[88618]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:03.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:02 smithi172 ceph-mon[67937]: pgmap v359: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:04.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:03 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:04.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:03 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:04.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:03 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:05.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:04 smithi164 ceph-mon[83622]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:05.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:04 smithi164 ceph-mon[88618]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:05.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:04 smithi172 ceph-mon[67937]: pgmap v360: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:06.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:05 smithi164 ceph-mon[83622]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:06.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:05 smithi164 ceph-mon[88618]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:06.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:05 smithi172 ceph-mon[67937]: pgmap v361: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:09.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:08 smithi164 ceph-mon[83622]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:09.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:08 smithi164 ceph-mon[88618]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:09.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:08 smithi172 ceph-mon[67937]: pgmap v362: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:10.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:10 smithi164 ceph-mon[83622]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:10.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:10 smithi164 ceph-mon[88618]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:11.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:10 smithi172 ceph-mon[67937]: pgmap v363: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:11.272 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ps' 2024-03-26T02:58:12.265 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:12 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:58:11] "GET /metrics HTTP/1.1" 200 38425 "" "Prometheus/2.43.0" 2024-03-26T02:58:13.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:12 smithi164 ceph-mon[83622]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:13.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:12 smithi164 ceph-mon[88618]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:13.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:12 smithi172 ceph-mon[67937]: pgmap v364: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:58:13.338 INFO:teuthology.orchestra.run.smithi164.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:alertmanager.a smithi164 *:9093,9094 running (12m) 2m ago 34m 26.0M - 0.25.0 c8568f914cd2 763628284e16 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:grafana.a smithi172 *:3000 running (12m) 2m ago 34m 80.9M - 9.4.12 2bacad6d85d8 c46cfc9279a0 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:iscsi.foo.smithi164.bueptz smithi164 running (13m) 2m ago 33m 58.7M - 3.5 e1d6a67b021e 043e1be14070 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.x smithi172 *:8443,9283,8765 running (13m) 2m ago 39m 454M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 fe45bef004bd 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:mgr.y smithi164 *:8443,9283,8765 running (24m) 2m ago 40m 515M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 09bd85f609f2 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:mon.a smithi164 running (10m) 2m ago 40m 58.9M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 94b7005f8327 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:mon.b smithi172 running (11m) 2m ago 39m 56.2M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 5de9fbe3024c 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:mon.c smithi164 running (10m) 2m ago 39m 48.5M 2048M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4909686c6b24 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.a smithi164 *:9100 running (13m) 2m ago 35m 16.0M - 1.5.0 0da6a335fe13 467a30dd8758 2024-03-26T02:58:13.339 INFO:teuthology.orchestra.run.smithi164.stdout:node-exporter.b smithi172 *:9100 running (13m) 2m ago 35m 16.1M - 1.5.0 0da6a335fe13 352ab9d69d2a 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:osd.0 smithi164 running (8m) 2m ago 38m 62.4M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 1b92fd14a23a 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:osd.1 smithi164 running (7m) 2m ago 38m 56.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fb9fd299fb0 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:osd.2 smithi164 running (6m) 2m ago 38m 61.7M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f4dde90188ad 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:osd.3 smithi164 running (6m) 2m ago 37m 60.6M 2992M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 724b53d6449c 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:osd.4 smithi172 running (5m) 2m ago 37m 64.0M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 4ca2e0eaa5bb 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:osd.5 smithi172 running (4m) 2m ago 37m 60.6M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 e40eb2ac9a0b 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:osd.6 smithi172 running (4m) 2m ago 36m 55.7M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 44e8c5cf078f 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:osd.7 smithi172 running (3m) 2m ago 36m 47.8M 3248M 19.0.0-2475-ge42b8ea3 f18c6672b5a3 66123d33bbf2 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:prometheus.a smithi172 *:9095 running (13m) 2m ago 35m 79.3M - 2.43.0 a07b618ecd1d 6e31603f6658 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi164.cdxqhg smithi164 *:8000 running (2m) 2m ago 34m 77.6M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 f7f6fade545f 2024-03-26T02:58:13.340 INFO:teuthology.orchestra.run.smithi164.stdout:rgw.foo.smithi172.gbbkmx smithi172 *:8000 running (2m) 2m ago 34m 75.3M - 19.0.0-2475-ge42b8ea3 f18c6672b5a3 7fc0285a1be3 2024-03-26T02:58:13.974 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:13 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:13.974 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:13 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:13.998 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions' 2024-03-26T02:58:14.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:13 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:15.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:14 smithi164 ceph-mon[83622]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:14 smithi164 ceph-mon[83622]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:15.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:14 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:14 smithi164 ceph-mon[88618]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:14 smithi164 ceph-mon[88618]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:15.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:14 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:14 smithi172 ceph-mon[67937]: from='client.54948 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:15.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:14 smithi172 ceph-mon[67937]: pgmap v365: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:15.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:14 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:16.156 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:58:16.156 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": { 2024-03-26T02:58:16.156 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 3 2024-03-26T02:58:16.156 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:58:16.156 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": { 2024-03-26T02:58:16.156 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:58:16.156 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": { 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 8 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw": { 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 2 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: "overall": { 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)": 15 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-03-26T02:58:16.157 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:58:16.737 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch upgrade status' 2024-03-26T02:58:17.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:16 smithi164 ceph-mon[88618]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:17.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:16 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2172542025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:17.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:16 smithi164 ceph-mon[83622]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:17.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:16 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2172542025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:17.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:16 smithi172 ceph-mon[67937]: pgmap v366: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:17.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:16 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2172542025' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:18.752 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:58:18.752 INFO:teuthology.orchestra.run.smithi164.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T02:58:18.752 INFO:teuthology.orchestra.run.smithi164.stdout: "in_progress": true, 2024-03-26T02:58:18.752 INFO:teuthology.orchestra.run.smithi164.stdout: "which": "Upgrading all daemon types on all hosts", 2024-03-26T02:58:18.752 INFO:teuthology.orchestra.run.smithi164.stdout: "services_complete": [ 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout: "rgw", 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout: "osd", 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr", 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout: "mon" 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout: "progress": "15/21 daemons upgraded", 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed.", 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout: "is_paused": true 2024-03-26T02:58:18.753 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:58:19.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:18 smithi164 ceph-mon[83622]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:19.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:18 smithi164 ceph-mon[88618]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:19.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:18 smithi172 ceph-mon[67937]: pgmap v367: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:19.368 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph health detail' 2024-03-26T02:58:19.940 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:19 smithi164 ceph-mon[83622]: from='client.64894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:19.941 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:19 smithi164 ceph-mon[88618]: from='client.64894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:20.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:19 smithi172 ceph-mon[67937]: from='client.64894 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:20.920 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:20 smithi164 ceph-mon[83622]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:20.920 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:20 smithi164 ceph-mon[88618]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:21.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:20 smithi172 ceph-mon[67937]: pgmap v368: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:21.417 INFO:teuthology.orchestra.run.smithi164.stdout:HEALTH_WARN Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. 2024-03-26T02:58:21.417 INFO:teuthology.orchestra.run.smithi164.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi164.bueptz on host smithi164 failed. 2024-03-26T02:58:21.417 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrade daemon: iscsi.foo.smithi164.bueptz: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi164.bueptz ... 2024-03-26T02:58:21.417 INFO:teuthology.orchestra.run.smithi164.stdout:Creating ceph-iscsi config... 2024-03-26T02:58:21.417 INFO:teuthology.orchestra.run.smithi164.stdout:Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/iscsi-gateway.cfg 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout:Write file: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/iscsi.foo.smithi164.bueptz/tcmu-runner-entrypoint.sh 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2de2654d3c\x2deb16\x2d11ee\x2d95d0\x2d87774f69a715.slice/ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout:Non-zero exit code 1 from systemctl start ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout:systemctl: stderr Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout:systemctl: stderr See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout:Traceback (most recent call last): 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout: return _run_code(code, main_globals, None, 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout: exec(code, run_globals) 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5608, in 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 5596, in main 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3096, in command_deploy_from 2024-03-26T02:58:21.418 INFO:teuthology.orchestra.run.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3131, in _common_deploy 2024-03-26T02:58:21.419 INFO:teuthology.orchestra.run.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 3151, in _deploy_daemon_container 2024-03-26T02:58:21.419 INFO:teuthology.orchestra.run.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1087, in deploy_daemon 2024-03-26T02:58:21.419 INFO:teuthology.orchestra.run.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/__main__.py", line 1223, in deploy_daemon_units 2024-03-26T02:58:21.419 INFO:teuthology.orchestra.run.smithi164.stdout: File "/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/cephadm.63d4c0df7d90915f33623a0e46a9539e31137c8c7e0b252b00c3fbd3cdb4490f/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-03-26T02:58:21.419 INFO:teuthology.orchestra.run.smithi164.stdout:RuntimeError: Failed command: systemctl start ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz: Job for ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service failed because the control process exited with error code. 2024-03-26T02:58:21.419 INFO:teuthology.orchestra.run.smithi164.stdout:See "systemctl status ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" and "journalctl -xeu ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@iscsi.foo.smithi164.bueptz.service" for details. 2024-03-26T02:58:22.159 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:22 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:58:22] "GET /metrics HTTP/1.1" 200 38426 "" "Prometheus/2.43.0" 2024-03-26T02:58:22.160 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:21 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2946259423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:58:22.160 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:21 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2946259423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:58:22.184 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-03-26T02:58:22.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:21 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2946259423' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-03-26T02:58:23.191 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:22 smithi164 ceph-mon[83622]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:23.191 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:22 smithi164 ceph-mon[88618]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:23.208 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:22 smithi172 ceph-mon[67937]: pgmap v369: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:24.023 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:23 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:24.023 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:23 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:24.294 INFO:teuthology.orchestra.run.smithi164.stdout:true 2024-03-26T02:58:24.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:23 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:24.900 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:24 smithi164 ceph-mon[83622]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:24.901 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:24 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/1281983217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:24.901 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:24 smithi164 ceph-mon[88618]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:24.901 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:24 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/1281983217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:24.923 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-03-26T02:58:25.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:24 smithi172 ceph-mon[67937]: pgmap v370: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:25.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:24 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/1281983217' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:27.000 INFO:teuthology.orchestra.run.smithi164.stdout: "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)" 2024-03-26T02:58:27.001 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:26 smithi164 ceph-mon[83622]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:27.001 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:26 smithi164 ceph-mon[88618]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:27.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:26 smithi172 ceph-mon[67937]: pgmap v371: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:27.639 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -e sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-03-26T02:58:27.941 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:27 smithi164 ceph-mon[88618]: from='client.? 172.21.15.164:0/2022967370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:27.941 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:27 smithi164 ceph-mon[83622]: from='client.? 172.21.15.164:0/2022967370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:28.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:27 smithi172 ceph-mon[67937]: from='client.? 172.21.15.164:0/2022967370' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-26T02:58:29.096 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:28 smithi164 ceph-mon[83622]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:29.097 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:28 smithi164 ceph-mon[88618]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:29.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:28 smithi172 ceph-mon[67937]: pgmap v372: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:29.514 INFO:teuthology.orchestra.run.smithi164.stdout:osd 8 2m ago - 2024-03-26T02:58:30.153 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:29 smithi164 ceph-mon[83622]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:30.154 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:29 smithi164 ceph-mon[88618]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:30.178 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-26T02:58:30.187 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi164.front.sepia.ceph.com 2024-03-26T02:58:30.187 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- bash -c 'ceph orch upgrade ls' 2024-03-26T02:58:30.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:29 smithi172 ceph-mon[67937]: from='mgr.25636 172.21.15.164:0/4047122601' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:58:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:30 smithi164 ceph-mon[83622]: from='client.64918 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:31.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:30 smithi164 ceph-mon[83622]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:30 smithi164 ceph-mon[88618]: from='client.64918 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:31.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:30 smithi164 ceph-mon[88618]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:30 smithi172 ceph-mon[67937]: from='client.64918 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:31.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:30 smithi172 ceph-mon[67937]: pgmap v373: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:32.294 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:32 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:58:32] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-03-26T02:58:32.747 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-03-26T02:58:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "image": "quay.io/ceph/ceph", 2024-03-26T02:58:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "registry": "quay.io", 2024-03-26T02:58:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "bare_image": "ceph/ceph", 2024-03-26T02:58:32.747 INFO:teuthology.orchestra.run.smithi164.stdout: "versions": [] 2024-03-26T02:58:32.748 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-03-26T02:58:33.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:32 smithi164 ceph-mon[83622]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:33.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:32 smithi164 ceph-mon[88618]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:33.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:32 smithi172 ceph-mon[67937]: pgmap v374: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:33.401 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-03-26T02:58:34.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:33 smithi164 ceph-mon[88618]: from='client.54990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:34.045 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:33 smithi164 ceph-mon[88618]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:34.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:33 smithi164 ceph-mon[83622]: from='client.54990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:34.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:33 smithi164 ceph-mon[83622]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:34.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:33 smithi172 ceph-mon[67937]: from='client.54990 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:34.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:33 smithi172 ceph-mon[67937]: from='client.54873 -' entity='client.iscsi.foo.smithi164.bueptz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:58:34.908 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:34 smithi164 ceph-mon[83622]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:34.908 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:34 smithi164 ceph-mon[88618]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:35.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:34 smithi172 ceph-mon[67937]: pgmap v375: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:35.948 INFO:teuthology.orchestra.run.smithi164.stdout: "16.2.0", 2024-03-26T02:58:36.506 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-03-26T02:58:37.094 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:36 smithi164 ceph-mon[83622]: from='client.64930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:37.094 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:36 smithi164 ceph-mon[83622]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:37.094 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:36 smithi164 ceph-mon[88618]: from='client.64930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:37.094 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:36 smithi164 ceph-mon[88618]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:37.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:36 smithi172 ceph-mon[67937]: from='client.64930 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:37.307 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:36 smithi172 ceph-mon[67937]: pgmap v376: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:58:39.086 INFO:teuthology.orchestra.run.smithi164.stdout: "v16.2.2", 2024-03-26T02:58:39.086 INFO:teuthology.orchestra.run.smithi164.stdout: "v16.2.2-20210505", 2024-03-26T02:58:39.294 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:38 smithi164 ceph-mon[88618]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:39.294 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:38 smithi164 ceph-mon[83622]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:39.306 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:38 smithi172 ceph-mon[67937]: pgmap v377: 161 pgs: 161 active+clean; 585 KiB data, 270 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:58:39.687 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T02:58:39.698 INFO:tasks.cephadm:Teardown begin 2024-03-26T02:58:39.698 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:58:39.729 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:58:39.765 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T02:58:39.765 DEBUG:teuthology.orchestra.run.smithi164:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:58:39.784 DEBUG:teuthology.orchestra.run.smithi172:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:58:39.822 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T02:58:39.823 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T02:58:39.823 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a 2024-03-26T02:58:40.044 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:39 smithi164 ceph-mon[88618]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:40.044 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:39 smithi164 ceph-mon[83622]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:40.045 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:39 smithi164 systemd[1]: Stopping Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:58:40.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:39 smithi172 ceph-mon[67937]: from='client.55002 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:58:40.510 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a[83618]: 2024-03-26T02:58:40.141+0000 7f3496e13700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:58:40.510 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a[83618]: 2024-03-26T02:58:40.141+0000 7f3496e13700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-03-26T02:58:40.510 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:40 smithi164 podman[129951]: 2024-03-26 02:58:40.213645779 +0000 UTC m=+0.198709477 container died 94b7005f8327c28c8df1c00c912fbfb64b7c89218646bf74a7e9c162a3eb9c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-26T02:58:40.510 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:58:40] ENGINE Bus STOPPING 2024-03-26T02:58:40.510 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:58:40] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:58:40.511 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:58:40] ENGINE Bus STOPPED 2024-03-26T02:58:40.511 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:58:40] ENGINE Bus STARTING 2024-03-26T02:58:40.511 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: CherryPy Checker: 2024-03-26T02:58:40.511 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: The Application mounted at '' has an empty config. 2024-03-26T02:58:40.511 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:58:40.511 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:58:40] ENGINE Serving on http://:::9283 2024-03-26T02:58:40.511 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:40 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: [26/Mar/2024:02:58:40] ENGINE Bus STARTED 2024-03-26T02:58:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:40 smithi164 podman[129951]: 2024-03-26 02:58:40.608916778 +0000 UTC m=+0.593980465 container cleanup 94b7005f8327c28c8df1c00c912fbfb64b7c89218646bf74a7e9c162a3eb9c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T02:58:40.794 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:40 smithi164 bash[129951]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a 2024-03-26T02:58:41.074 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T02:58:41.099 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:40 smithi164 podman[129977]: 2024-03-26 02:58:40.835202425 +0000 UTC m=+0.619744871 container remove 94b7005f8327c28c8df1c00c912fbfb64b7c89218646bf74a7e9c162a3eb9c7b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T02:58:41.099 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:41 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T02:58:41.100 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:41 smithi164 systemd[1]: Stopped Ceph mon.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:58:41.100 INFO:journalctl@ceph.mon.a.smithi164.stdout:Mar 26 02:58:41 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.a.service: Consumed 10.319s CPU time. 2024-03-26T02:58:41.127 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:58:41.127 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T02:58:41.127 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T02:58:41.127 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.c 2024-03-26T02:58:41.455 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:41 smithi164 systemd[1]: Stopping Ceph mon.c for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:58:41.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c[88614]: 2024-03-26T02:58:41.454+0000 7f48c9167700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:58:41.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:41 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c[88614]: 2024-03-26T02:58:41.454+0000 7f48c9167700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-03-26T02:58:41.794 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:41 smithi164 podman[130084]: 2024-03-26 02:58:41.547000307 +0000 UTC m=+0.195738189 container died 4909686c6b24901873cb1c278d6da7dbfce2c218eb2508470a84fb599453677c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T02:58:42.208 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:42 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: ::ffff:172.21.15.172 - - [26/Mar/2024:02:58:42] "GET /metrics HTTP/1.1" 200 38422 "" "Prometheus/2.43.0" 2024-03-26T02:58:42.208 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:41 smithi164 podman[130084]: 2024-03-26 02:58:41.965152194 +0000 UTC m=+0.613890069 container cleanup 4909686c6b24901873cb1c278d6da7dbfce2c218eb2508470a84fb599453677c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T02:58:42.208 INFO:journalctl@ceph.mon.c.smithi164.stdout:Mar 26 02:58:41 smithi164 bash[130084]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-c 2024-03-26T02:58:42.333 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T02:58:42.369 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:58:42.369 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T02:58:42.369 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T02:58:42.369 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.b 2024-03-26T02:58:42.757 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:42 smithi172 systemd[1]: Stopping Ceph mon.b for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:58:43.056 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b[67933]: 2024-03-26T02:58:42.755+0000 7f1e682a1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:58:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b[67933]: 2024-03-26T02:58:42.755+0000 7f1e682a1700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-03-26T02:58:43.057 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:42 smithi172 podman[102027]: 2024-03-26 02:58:42.848709572 +0000 UTC m=+0.288492246 container died 5de9fbe3024c5677f80b4d92f9c3b0c9d976e111aed54f3cac40ddc01064dfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:58:43.556 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:43 smithi172 podman[102027]: 2024-03-26 02:58:43.315082239 +0000 UTC m=+0.754864925 container cleanup 5de9fbe3024c5677f80b4d92f9c3b0c9d976e111aed54f3cac40ddc01064dfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T02:58:43.557 INFO:journalctl@ceph.mon.b.smithi172.stdout:Mar 26 02:58:43 smithi172 bash[102027]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mon-b 2024-03-26T02:58:43.783 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T02:58:43.829 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:58:43.830 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T02:58:43.830 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T02:58:43.830 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y 2024-03-26T02:58:44.120 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:43 smithi164 systemd[1]: Stopping Ceph mgr.y for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:58:44.396 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:58:44.118+0000 7f6e1d90c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:58:44.397 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y[62326]: 2024-03-26T02:58:44.118+0000 7f6e1d90c700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-26T02:58:44.397 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 podman[130213]: 2024-03-26 02:58:44.160979437 +0000 UTC m=+0.149173889 container died 09bd85f609f26538ecc629d423f3b62b06d695d3cb243535ce7fa58360328aa3 (image=quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T02:58:44.738 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 podman[130213]: 2024-03-26 02:58:44.494330589 +0000 UTC m=+0.482525030 container cleanup 09bd85f609f26538ecc629d423f3b62b06d695d3cb243535ce7fa58360328aa3 (image=quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=) 2024-03-26T02:58:44.738 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 bash[130213]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y 2024-03-26T02:58:44.976 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T02:58:45.001 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 podman[130229]: 2024-03-26 02:58:44.737694432 +0000 UTC m=+0.580996859 container remove 09bd85f609f26538ecc629d423f3b62b06d695d3cb243535ce7fa58360328aa3 (image=quay.ceph.io/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:58:45.001 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service: Deactivated successfully. 2024-03-26T02:58:45.001 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 systemd[1]: Stopped Ceph mgr.y for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:58:45.001 INFO:journalctl@ceph.mgr.y.smithi164.stdout:Mar 26 02:58:44 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.y.service: Consumed 1min 12.756s CPU time. 2024-03-26T02:58:45.016 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:58:45.016 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T02:58:45.016 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T02:58:45.016 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x 2024-03-26T02:58:45.306 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:58:45 smithi172 systemd[1]: Stopping Ceph mgr.x for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:58:45.695 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:58:45 smithi172 podman[102147]: 2024-03-26 02:58:45.360095861 +0000 UTC m=+0.146314733 container died fe45bef004bdcc5389f2e5408df19ca8f6ecff4169f5144d9b407916dd4f9b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T02:58:45.695 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:58:45 smithi172 podman[102147]: 2024-03-26 02:58:45.694545621 +0000 UTC m=+0.480764498 container cleanup fe45bef004bdcc5389f2e5408df19ca8f6ecff4169f5144d9b407916dd4f9b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:58:46.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:58:45 smithi172 bash[102147]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x 2024-03-26T02:58:46.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:58:45 smithi172 podman[102161]: 2024-03-26 02:58:45.904435256 +0000 UTC m=+0.541906479 container remove fe45bef004bdcc5389f2e5408df19ca8f6ecff4169f5144d9b407916dd4f9b92 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T02:58:46.057 INFO:journalctl@ceph.mgr.x.smithi172.stdout:Mar 26 02:58:45 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:58:46.107 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T02:58:46.144 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:58:46.144 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T02:58:46.144 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T02:58:46.144 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.0 2024-03-26T02:58:46.424 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:46 smithi164 systemd[1]: Stopping Ceph osd.0 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:58:46.794 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:46 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[95998]: 2024-03-26T02:58:46.423+0000 7fdda9482700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:58:46.794 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:46 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[95998]: 2024-03-26T02:58:46.423+0000 7fdda9482700 -1 osd.0 159 *** Got signal Terminated *** 2024-03-26T02:58:46.794 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:46 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0[95998]: 2024-03-26T02:58:46.423+0000 7fdda9482700 -1 osd.0 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:58:52.216 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:51 smithi164 podman[130337]: 2024-03-26 02:58:51.895618966 +0000 UTC m=+5.565510625 container died 1b92fd14a23a69fbb4f037226ac3bcda32850bd0dfdbe8fd389fd822f55a1fa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:58:52.216 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:52 smithi164 podman[130337]: 2024-03-26 02:58:52.215526545 +0000 UTC m=+5.885418268 container cleanup 1b92fd14a23a69fbb4f037226ac3bcda32850bd0dfdbe8fd389fd822f55a1fa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T02:58:52.544 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:52 smithi164 bash[130337]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0 2024-03-26T02:58:52.988 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:52 smithi164 podman[130354]: 2024-03-26 02:58:52.574899668 +0000 UTC m=+0.672990453 container remove 1b92fd14a23a69fbb4f037226ac3bcda32850bd0dfdbe8fd389fd822f55a1fa0 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T02:58:53.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:52 smithi164 podman[130448]: 2024-03-26 02:58:52.891064973 +0000 UTC m=+0.018030494 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:58:53.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:52 smithi164 podman[130448]: 2024-03-26 02:58:52.992504324 +0000 UTC m=+0.119469831 container create c05dbc41c84e6a4e457851fa9bf3fc782bda47e69f5a5e580e119e465726c47b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS) 2024-03-26T02:58:53.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:53 smithi164 podman[130448]: 2024-03-26 02:58:53.203410596 +0000 UTC m=+0.330376104 container init c05dbc41c84e6a4e457851fa9bf3fc782bda47e69f5a5e580e119e465726c47b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T02:58:53.295 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:53 smithi164 podman[130448]: 2024-03-26 02:58:53.206677851 +0000 UTC m=+0.333643361 container start c05dbc41c84e6a4e457851fa9bf3fc782bda47e69f5a5e580e119e465726c47b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-26T02:58:53.782 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:53 smithi164 podman[130448]: 2024-03-26 02:58:53.303344296 +0000 UTC m=+0.430309818 container attach c05dbc41c84e6a4e457851fa9bf3fc782bda47e69f5a5e580e119e465726c47b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2) 2024-03-26T02:58:53.782 INFO:journalctl@ceph.osd.0.smithi164.stdout:Mar 26 02:58:53 smithi164 podman[130448]: 2024-03-26 02:58:53.501070788 +0000 UTC m=+0.628036297 container died c05dbc41c84e6a4e457851fa9bf3fc782bda47e69f5a5e580e119e465726c47b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2) 2024-03-26T02:58:54.093 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T02:58:54.138 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:58:54.138 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T02:58:54.139 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T02:58:54.139 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.1 2024-03-26T02:58:54.513 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:58:54 smithi164 systemd[1]: Stopping Ceph osd.1 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:58:54.794 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:58:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[100942]: 2024-03-26T02:58:54.510+0000 7f9c8e75b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:58:54.794 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:58:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[100942]: 2024-03-26T02:58:54.510+0000 7f9c8e75b700 -1 osd.1 159 *** Got signal Terminated *** 2024-03-26T02:58:54.794 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:58:54 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1[100942]: 2024-03-26T02:58:54.510+0000 7f9c8e75b700 -1 osd.1 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:59:00.194 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:58:59 smithi164 podman[130558]: 2024-03-26 02:58:59.90962151 +0000 UTC m=+5.518543867 container died 7fb9fd299fb06725571330d0a774a87cb0f87596d9773789971c32ea9cf09ce3 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:59:00.535 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:00 smithi164 podman[130558]: 2024-03-26 02:59:00.292912818 +0000 UTC m=+5.901835175 container cleanup 7fb9fd299fb06725571330d0a774a87cb0f87596d9773789971c32ea9cf09ce3 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:59:00.535 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:00 smithi164 bash[130558]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1 2024-03-26T02:59:00.794 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:00 smithi164 podman[130576]: 2024-03-26 02:59:00.535599137 +0000 UTC m=+0.624657192 container remove 7fb9fd299fb06725571330d0a774a87cb0f87596d9773789971c32ea9cf09ce3 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:59:01.124 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:00 smithi164 podman[130666]: 2024-03-26 02:59:00.767660952 +0000 UTC m=+0.018372604 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:59:01.124 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:00 smithi164 podman[130666]: 2024-03-26 02:59:00.887083362 +0000 UTC m=+0.137795022 container create ee2327a366da92f662ebb48e54bbecf273a6e319ebabf604304846723806fa3d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-26T02:59:01.124 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:01 smithi164 podman[130666]: 2024-03-26 02:59:01.123171401 +0000 UTC m=+0.373883026 container init ee2327a366da92f662ebb48e54bbecf273a6e319ebabf604304846723806fa3d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T02:59:01.415 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:01 smithi164 podman[130666]: 2024-03-26 02:59:01.126286629 +0000 UTC m=+0.376998256 container start ee2327a366da92f662ebb48e54bbecf273a6e319ebabf604304846723806fa3d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T02:59:01.415 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:01 smithi164 podman[130666]: 2024-03-26 02:59:01.206400278 +0000 UTC m=+0.457111928 container attach ee2327a366da92f662ebb48e54bbecf273a6e319ebabf604304846723806fa3d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T02:59:01.676 INFO:journalctl@ceph.osd.1.smithi164.stdout:Mar 26 02:59:01 smithi164 podman[130692]: 2024-03-26 02:59:01.440602719 +0000 UTC m=+0.019130216 container died ee2327a366da92f662ebb48e54bbecf273a6e319ebabf604304846723806fa3d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:59:01.961 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T02:59:02.005 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:59:02.006 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T02:59:02.006 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T02:59:02.006 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.2 2024-03-26T02:59:02.294 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:02 smithi164 systemd[1]: Stopping Ceph osd.2 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:02.794 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[107148]: 2024-03-26T02:59:02.333+0000 7f532f3db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:59:02.794 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[107148]: 2024-03-26T02:59:02.333+0000 7f532f3db700 -1 osd.2 159 *** Got signal Terminated *** 2024-03-26T02:59:02.794 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:02 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2[107148]: 2024-03-26T02:59:02.333+0000 7f532f3db700 -1 osd.2 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:59:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:04.683Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.164:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.164:8765: connect: connection refused" 2024-03-26T02:59:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:04.686Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.164:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.164:8765: connect: connection refused" 2024-03-26T02:59:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:04.687Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.164:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.164:8765: connect: connection refused" 2024-03-26T02:59:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:04.687Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.164:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.164:8765: connect: connection refused" 2024-03-26T02:59:05.057 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:04 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:04.688Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.164:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.164:8765: connect: connection refused" 2024-03-26T02:59:07.938 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:07 smithi164 podman[130770]: 2024-03-26 02:59:07.664950089 +0000 UTC m=+5.440087271 container died f4dde90188ad9316bc905687072e917b96fbb74b39e767fc14eeb61f65f70454 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:59:08.294 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:08 smithi164 podman[130770]: 2024-03-26 02:59:08.03697442 +0000 UTC m=+5.812111585 container cleanup f4dde90188ad9316bc905687072e917b96fbb74b39e767fc14eeb61f65f70454 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:59:08.294 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:08 smithi164 bash[130770]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2 2024-03-26T02:59:08.665 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:08 smithi164 podman[130788]: 2024-03-26 02:59:08.338204148 +0000 UTC m=+0.669866315 container remove f4dde90188ad9316bc905687072e917b96fbb74b39e767fc14eeb61f65f70454 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T02:59:08.941 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:08 smithi164 podman[130881]: 2024-03-26 02:59:08.568190682 +0000 UTC m=+0.018246706 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:59:08.941 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:08 smithi164 podman[130881]: 2024-03-26 02:59:08.67234637 +0000 UTC m=+0.122402429 container create 55d7981b847472d62ce1a7045bbdb593131b435ec8f9d6f88293324238bcd317 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0) 2024-03-26T02:59:08.941 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:08 smithi164 podman[130881]: 2024-03-26 02:59:08.882828029 +0000 UTC m=+0.332884045 container init 55d7981b847472d62ce1a7045bbdb593131b435ec8f9d6f88293324238bcd317 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T02:59:08.942 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:08 smithi164 podman[130881]: 2024-03-26 02:59:08.885895664 +0000 UTC m=+0.335951683 container start 55d7981b847472d62ce1a7045bbdb593131b435ec8f9d6f88293324238bcd317 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T02:59:08.942 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:08 smithi164 podman[130881]: 2024-03-26 02:59:08.941291668 +0000 UTC m=+0.391347690 container attach 55d7981b847472d62ce1a7045bbdb593131b435ec8f9d6f88293324238bcd317 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-03-26T02:59:09.294 INFO:journalctl@ceph.osd.2.smithi164.stdout:Mar 26 02:59:09 smithi164 podman[130881]: 2024-03-26 02:59:09.177475134 +0000 UTC m=+0.627531151 container died 55d7981b847472d62ce1a7045bbdb593131b435ec8f9d6f88293324238bcd317 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T02:59:09.789 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T02:59:09.833 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:59:09.833 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T02:59:09.833 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T02:59:09.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.3 2024-03-26T02:59:10.044 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:09 smithi164 systemd[1]: Stopping Ceph osd.3 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:10.544 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:59:10.181+0000 7fbe95064700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:59:10.544 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:59:10.181+0000 7fbe95064700 -1 osd.3 159 *** Got signal Terminated *** 2024-03-26T02:59:10.544 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:10 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:59:10.181+0000 7fbe95064700 -1 osd.3 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:59:12.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:12.121+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:13.306 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:13.102+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:13.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:12 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:12.870+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:14.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:13.567Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi164.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:59:14.045 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:13 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:13.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-26T02:59:14.057 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:13.741+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:14.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:13 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:13.905+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:14.461 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:14 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:59:14.213+0000 7fbe8d650700 -1 osd.3 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:45.618228+0000 front 2024-03-26T02:58:45.618314+0000 (oldest deadline 2024-03-26T02:59:13.218850+0000) 2024-03-26T02:59:14.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:14.107+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:14.986 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:14.697+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:14.986 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:14.893+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:15.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:14 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:14.985+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:15.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:15.114+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:15.544 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:15 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3[113547]: 2024-03-26T02:59:15.167+0000 7fbe8d650700 -1 osd.3 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:45.618228+0000 front 2024-03-26T02:58:45.618314+0000 (oldest deadline 2024-03-26T02:59:13.218850+0000) 2024-03-26T02:59:15.832 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:15 smithi164 podman[130986]: 2024-03-26 02:59:15.554854834 +0000 UTC m=+5.496717983 container died 724b53d6449c2f4f1d97c56335f2318de80ffeea2a3a76ed538feb232ad5818d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319) 2024-03-26T02:59:15.963 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:15.694+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:15.963 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:15.888+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:16.140 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:15 smithi164 podman[130986]: 2024-03-26 02:59:15.930655578 +0000 UTC m=+5.872518722 container cleanup 724b53d6449c2f4f1d97c56335f2318de80ffeea2a3a76ed538feb232ad5818d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:59:16.140 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:15 smithi164 bash[130986]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3 2024-03-26T02:59:16.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:16.092+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:16.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:15 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:15.962+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:16.455 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:16 smithi164 podman[131015]: 2024-03-26 02:59:16.140113426 +0000 UTC m=+0.587624795 container remove 724b53d6449c2f4f1d97c56335f2318de80ffeea2a3a76ed538feb232ad5818d (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T02:59:16.794 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:16 smithi164 podman[131105]: 2024-03-26 02:59:16.357911718 +0000 UTC m=+0.018174352 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:59:16.794 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:16 smithi164 podman[131105]: 2024-03-26 02:59:16.479529156 +0000 UTC m=+0.139791796 container create 392cce065ac61d2c6d2e82d4a8b848dbf3e32fc7dce8718670c64cff3a76159b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:59:16.794 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:16 smithi164 podman[131105]: 2024-03-26 02:59:16.701720745 +0000 UTC m=+0.361983393 container init 392cce065ac61d2c6d2e82d4a8b848dbf3e32fc7dce8718670c64cff3a76159b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T02:59:16.795 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:16 smithi164 podman[131105]: 2024-03-26 02:59:16.705125259 +0000 UTC m=+0.365387893 container start 392cce065ac61d2c6d2e82d4a8b848dbf3e32fc7dce8718670c64cff3a76159b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:59:16.995 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:16.653+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:16.995 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:16.855+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:17.294 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:16 smithi164 podman[131105]: 2024-03-26 02:59:16.821368944 +0000 UTC m=+0.481631587 container attach 392cce065ac61d2c6d2e82d4a8b848dbf3e32fc7dce8718670c64cff3a76159b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:59:17.294 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:16 smithi164 podman[131105]: 2024-03-26 02:59:16.997481761 +0000 UTC m=+0.657744404 container died 392cce065ac61d2c6d2e82d4a8b848dbf3e32fc7dce8718670c64cff3a76159b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:59:17.306 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:17.126+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:17.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:16 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:16.993+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:17.593 INFO:journalctl@ceph.osd.3.smithi164.stdout:Mar 26 02:59:17 smithi164 podman[131131]: 2024-03-26 02:59:17.57133572 +0000 UTC m=+0.568756861 container remove 392cce065ac61d2c6d2e82d4a8b848dbf3e32fc7dce8718670c64cff3a76159b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T02:59:17.599 DEBUG:teuthology.orchestra.run.smithi164:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T02:59:17.642 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:59:17.642 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T02:59:17.642 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T02:59:17.642 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.4 2024-03-26T02:59:17.938 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:17 smithi172 systemd[1]: Stopping Ceph osd.4 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:17.939 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:17.664+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:17.939 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:17.835+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:18.306 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:18.042+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:18.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:17.937+0000 7f90757bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:59:18.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:17.937+0000 7f90757bc700 -1 osd.4 159 *** Got signal Terminated *** 2024-03-26T02:59:18.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:17 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:17.937+0000 7f90757bc700 -1 osd.4 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:59:18.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:18.131+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:19.030 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:18.689+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:19.031 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:18 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:18.865+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:19.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:19.168+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:19.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:19.030+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:20.028 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:19.652+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:20.028 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:19 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:19.858+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:20.306 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:20.125+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:20.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:20.027+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:20.857 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:20.605+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:21.176 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:20.981+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:21.176 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:20.981+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:21.176 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:20.857+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:21.176 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:20.857+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:21.556 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:21.175+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:21.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:21.175+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.604021+0000 front 2024-03-26T02:58:55.603977+0000 (oldest deadline 2024-03-26T02:59:20.903947+0000) 2024-03-26T02:59:21.898 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:21.600+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:21.898 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:21.600+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:22.152 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:21.897+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:22.152 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:21.897+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:22.152 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:21.934+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:22.152 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:21 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:21.934+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:22.556 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:22.151+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:46.203240+0000 front 2024-03-26T02:58:46.203262+0000 (oldest deadline 2024-03-26T02:59:12.103150+0000) 2024-03-26T02:59:22.557 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4[78864]: 2024-03-26T02:59:22.151+0000 7f906dda8700 -1 osd.4 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.604021+0000 front 2024-03-26T02:58:55.603977+0000 (oldest deadline 2024-03-26T02:59:20.903947+0000) 2024-03-26T02:59:22.968 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:22.645+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:22.968 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:22.645+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:22.969 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:22.867+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:22.969 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:22.867+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:23.306 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:22.967+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:23.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:22 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:22.967+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:23.632 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:23 smithi172 podman[102285]: 2024-03-26 02:59:23.329951619 +0000 UTC m=+5.489204681 container died 4ca2e0eaa5bbbdc162fe802dca57c2a7a5803fb1c8b921dcf146b4d1db9bbd25 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T02:59:23.956 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:23 smithi172 podman[102285]: 2024-03-26 02:59:23.729760801 +0000 UTC m=+5.889013884 container cleanup 4ca2e0eaa5bbbdc162fe802dca57c2a7a5803fb1c8b921dcf146b4d1db9bbd25 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T02:59:23.957 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:23 smithi172 bash[102285]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4 2024-03-26T02:59:23.957 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:23.631+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:23.957 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:23.631+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:23.957 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:23.855+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:23.957 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:23.855+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:24.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:23.567Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=7 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi164.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:59:24.045 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:23.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi164.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:59:24.045 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:23 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:23.568Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-26T02:59:24.307 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:24 smithi172 podman[102302]: 2024-03-26 02:59:24.022953713 +0000 UTC m=+0.687844292 container remove 4ca2e0eaa5bbbdc162fe802dca57c2a7a5803fb1c8b921dcf146b4d1db9bbd25 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T02:59:24.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:23.955+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:24.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:23 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:23.955+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:24.596 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:24 smithi172 podman[102399]: 2024-03-26 02:59:24.257551881 +0000 UTC m=+0.018606521 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:59:24.596 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:24 smithi172 podman[102399]: 2024-03-26 02:59:24.365557222 +0000 UTC m=+0.126611890 container create 12c909c375075305a22ea05d16855d6785508de96c973ea5373e20e35d88e876 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319) 2024-03-26T02:59:24.895 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:24.845+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:24.895 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:24.845+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:24.895 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:24.595+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:24.896 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:24.595+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:24.896 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:24 smithi172 podman[102399]: 2024-03-26 02:59:24.60236971 +0000 UTC m=+0.363424347 container init 12c909c375075305a22ea05d16855d6785508de96c973ea5373e20e35d88e876 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:59:24.896 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:24 smithi172 podman[102399]: 2024-03-26 02:59:24.605557254 +0000 UTC m=+0.366611886 container start 12c909c375075305a22ea05d16855d6785508de96c973ea5373e20e35d88e876 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T02:59:24.896 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:24 smithi172 podman[102399]: 2024-03-26 02:59:24.677313746 +0000 UTC m=+0.438368383 container attach 12c909c375075305a22ea05d16855d6785508de96c973ea5373e20e35d88e876 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T02:59:25.181 INFO:journalctl@ceph.osd.4.smithi172.stdout:Mar 26 02:59:24 smithi172 podman[102399]: 2024-03-26 02:59:24.895193 +0000 UTC m=+0.656247625 container died 12c909c375075305a22ea05d16855d6785508de96c973ea5373e20e35d88e876 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T02:59:25.181 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:24.995+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:25.181 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:24 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:24.995+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:25.582 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T02:59:25.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:59:25.624 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T02:59:25.624 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T02:59:25.624 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.5 2024-03-26T02:59:25.806 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:25.566+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:25.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:25.566+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:25.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:25 smithi172 systemd[1]: Stopping Ceph osd.5 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:26.307 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:25.983+0000 7f48f7ebb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:59:26.307 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:25.983+0000 7f48f7ebb700 -1 osd.5 159 *** Got signal Terminated *** 2024-03-26T02:59:26.307 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:25.983+0000 7f48f7ebb700 -1 osd.5 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:59:26.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:25.976+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:26.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:25.976+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:26.308 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:25.854+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:26.308 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:25 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:25.854+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:26.894 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:26 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:26.584+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:26.894 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:26 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:26.584+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:27.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:26 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:26.892+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:27.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:26 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:26.892+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:27.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:27.024+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:27.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:27.024+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:27.915 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:27.559+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:27.915 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:27.559+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:28.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:27.914+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:28.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:27.914+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:28.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:27.977+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:28.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:27 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:27.977+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:28.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:28.551+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:28.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:28.551+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:28.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:28.551+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:03.234496+0000 front 2024-03-26T02:59:03.234636+0000 (oldest deadline 2024-03-26T02:59:27.934506+0000) 2024-03-26T02:59:29.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:28.951+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:29.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:28.951+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:29.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:28.952+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:29.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:28.952+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:29.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:28 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:28.952+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:29.985 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:29.577+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:29.985 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:29.577+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:29.985 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:29.577+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:03.234496+0000 front 2024-03-26T02:59:03.234636+0000 (oldest deadline 2024-03-26T02:59:27.934506+0000) 2024-03-26T02:59:30.306 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:29.984+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:30.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:29.984+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:30.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:29.988+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:30.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:29.988+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:30.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:29 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:29.988+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:30.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:30.551+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:51.132319+0000 front 2024-03-26T02:58:51.132204+0000 (oldest deadline 2024-03-26T02:59:12.832222+0000) 2024-03-26T02:59:30.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:30.551+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:58.733306+0000 front 2024-03-26T02:58:58.733322+0000 (oldest deadline 2024-03-26T02:59:20.933539+0000) 2024-03-26T02:59:30.807 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:30 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5[83847]: 2024-03-26T02:59:30.551+0000 7f48f04a7700 -1 osd.5 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:03.234496+0000 front 2024-03-26T02:59:03.234636+0000 (oldest deadline 2024-03-26T02:59:27.934506+0000) 2024-03-26T02:59:31.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:31.006+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:31.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:31.006+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:31.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:31.008+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:31.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:31.008+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:31.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:31.008+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:31.619 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:31 smithi172 podman[102505]: 2024-03-26 02:59:31.352158019 +0000 UTC m=+5.485520734 container died e40eb2ac9a0b8e3a5d5eb2fd063c418885465bb8abdb7605e3c4fbd704e79e27 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0) 2024-03-26T02:59:31.952 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:31 smithi172 podman[102505]: 2024-03-26 02:59:31.767911431 +0000 UTC m=+5.901274134 container cleanup e40eb2ac9a0b8e3a5d5eb2fd063c418885465bb8abdb7605e3c4fbd704e79e27 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T02:59:31.952 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:31 smithi172 bash[102505]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5 2024-03-26T02:59:32.306 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:31 smithi172 podman[102534]: 2024-03-26 02:59:31.952417133 +0000 UTC m=+0.600752024 container remove e40eb2ac9a0b8e3a5d5eb2fd063c418885465bb8abdb7605e3c4fbd704e79e27 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2) 2024-03-26T02:59:32.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:32 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:32.053+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:32.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:32 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:32.053+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:32.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:32 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:32.053+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:32.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:31.993+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:32.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:31.993+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:32.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:31 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:31.993+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:32.639 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:32 smithi172 podman[102628]: 2024-03-26 02:59:32.285127757 +0000 UTC m=+0.017756463 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:59:32.639 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:32 smithi172 podman[102628]: 2024-03-26 02:59:32.395245971 +0000 UTC m=+0.127874673 container create 1a53b1c2fa3e8359c60982900537cf71d5554e03d1c2834fc4f3c3400fe8560c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T02:59:32.934 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:32 smithi172 podman[102628]: 2024-03-26 02:59:32.639314653 +0000 UTC m=+0.371943355 container init 1a53b1c2fa3e8359c60982900537cf71d5554e03d1c2834fc4f3c3400fe8560c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T02:59:32.934 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:32 smithi172 podman[102628]: 2024-03-26 02:59:32.642445792 +0000 UTC m=+0.375074500 container start 1a53b1c2fa3e8359c60982900537cf71d5554e03d1c2834fc4f3c3400fe8560c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5) 2024-03-26T02:59:32.934 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:32 smithi172 podman[102628]: 2024-03-26 02:59:32.722613767 +0000 UTC m=+0.455242485 container attach 1a53b1c2fa3e8359c60982900537cf71d5554e03d1c2834fc4f3c3400fe8560c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T02:59:33.226 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:32 smithi172 podman[102628]: 2024-03-26 02:59:32.931636223 +0000 UTC m=+0.664264925 container died 1a53b1c2fa3e8359c60982900537cf71d5554e03d1c2834fc4f3c3400fe8560c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T02:59:33.226 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:32 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:32.994+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:33.226 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:32 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:32.994+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:33.226 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:32 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:32.994+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:33.226 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:33.064+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:33.227 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:33.064+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:33.227 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:33.064+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:33.486 INFO:journalctl@ceph.osd.5.smithi172.stdout:Mar 26 02:59:33 smithi172 podman[102654]: 2024-03-26 02:59:33.467936262 +0000 UTC m=+0.531007006 container remove 1a53b1c2fa3e8359c60982900537cf71d5554e03d1c2834fc4f3c3400fe8560c (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T02:59:33.491 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T02:59:33.532 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:59:33.533 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T02:59:33.533 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T02:59:33.533 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.6 2024-03-26T02:59:33.806 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:33 smithi172 systemd[1]: Stopping Ceph osd.6 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:34.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:33.568Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi164.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:59:34.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:33.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi164.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:59:34.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:33 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:33.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-26T02:59:34.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:34.090+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:34.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:34.090+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:34.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:34.090+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:34.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:33.879+0000 7f7c27762700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:59:34.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:33.879+0000 7f7c27762700 -1 osd.6 159 *** Got signal Terminated *** 2024-03-26T02:59:34.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:33.879+0000 7f7c27762700 -1 osd.6 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:59:34.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:33.945+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:34.308 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:33.945+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:34.308 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:33 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:33.945+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:35.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:34.967+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:35.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:34.968+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:35.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:34 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:34.968+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:35.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:35.127+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:35.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:35.127+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:35.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:35.127+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:36.249 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:36.172+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:36.249 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:36.172+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:36.249 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:36.172+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:36.249 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:35.954+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:36.249 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:35.954+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:36.249 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:35 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:35.954+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:37.161 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:36.986+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:37.162 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:36.986+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:37.162 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:36.986+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:37.162 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:36 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:36.986+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:10.399921+0000 front 2024-03-26T02:59:10.399890+0000 (oldest deadline 2024-03-26T02:59:36.299777+0000) 2024-03-26T02:59:37.556 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:37.160+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:37.557 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:37.160+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:37.557 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:37.160+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:38.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:37.940+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:49.996876+0000 front 2024-03-26T02:58:49.996988+0000 (oldest deadline 2024-03-26T02:59:14.096951+0000) 2024-03-26T02:59:38.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:37.940+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:55.197997+0000 front 2024-03-26T02:58:55.197901+0000 (oldest deadline 2024-03-26T02:59:20.497956+0000) 2024-03-26T02:59:38.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:37.940+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:05.798967+0000 front 2024-03-26T02:59:05.798899+0000 (oldest deadline 2024-03-26T02:59:28.698878+0000) 2024-03-26T02:59:38.307 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:37 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6[88428]: 2024-03-26T02:59:37.940+0000 7f7c1fd4e700 -1 osd.6 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:10.399921+0000 front 2024-03-26T02:59:10.399890+0000 (oldest deadline 2024-03-26T02:59:36.299777+0000) 2024-03-26T02:59:38.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:38.113+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:38.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:38.113+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:38.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:38.113+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:38.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:38 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:38.113+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:39.406 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:39.106+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:39.406 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:39.106+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:39.406 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:39.106+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:39.406 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:39 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:39.106+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:39.688 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:39 smithi172 podman[102733]: 2024-03-26 02:59:39.405709299 +0000 UTC m=+5.655170377 container died 44e8c5cf078f8fae118fdf3d76a92be3e9c9c4cfa30eb9bae0d4a3a91d3a8e42 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux ) 2024-03-26T02:59:40.057 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:39 smithi172 podman[102733]: 2024-03-26 02:59:39.862386947 +0000 UTC m=+6.111848075 container cleanup 44e8c5cf078f8fae118fdf3d76a92be3e9c9c4cfa30eb9bae0d4a3a91d3a8e42 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T02:59:40.057 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:39 smithi172 bash[102733]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6 2024-03-26T02:59:40.474 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:40 smithi172 podman[102750]: 2024-03-26 02:59:40.155359129 +0000 UTC m=+0.740223765 container remove 44e8c5cf078f8fae118fdf3d76a92be3e9c9c4cfa30eb9bae0d4a3a91d3a8e42 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:59:40.474 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:40.068+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:40.474 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:40.069+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:40.474 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:40.069+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:40.474 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:40 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:40.069+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:40.767 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:40 smithi172 podman[102840]: 2024-03-26 02:59:40.376687948 +0000 UTC m=+0.017881212 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:59:40.767 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:40 smithi172 podman[102840]: 2024-03-26 02:59:40.481145035 +0000 UTC m=+0.122338292 container create 63ed065467f8f3e943197afe900dce55773bf059c4fef0f0d0bb25715a8d7f9b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:59:40.767 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:40 smithi172 podman[102840]: 2024-03-26 02:59:40.700602871 +0000 UTC m=+0.341796128 container init 63ed065467f8f3e943197afe900dce55773bf059c4fef0f0d0bb25715a8d7f9b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T02:59:40.768 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:40 smithi172 podman[102840]: 2024-03-26 02:59:40.703744535 +0000 UTC m=+0.344937795 container start 63ed065467f8f3e943197afe900dce55773bf059c4fef0f0d0bb25715a8d7f9b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-26T02:59:40.768 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:40 smithi172 podman[102840]: 2024-03-26 02:59:40.767238238 +0000 UTC m=+0.408431512 container attach 63ed065467f8f3e943197afe900dce55773bf059c4fef0f0d0bb25715a8d7f9b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T02:59:41.041 INFO:journalctl@ceph.osd.6.smithi172.stdout:Mar 26 02:59:40 smithi172 podman[102840]: 2024-03-26 02:59:40.995501226 +0000 UTC m=+0.636694478 container died 63ed065467f8f3e943197afe900dce55773bf059c4fef0f0d0bb25715a8d7f9b (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T02:59:41.296 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:41.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:41.296 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:41.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:41.296 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:41.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:41.296 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:41.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:41.577 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T02:59:41.620 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:59:41.620 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T02:59:41.620 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T02:59:41.621 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.7 2024-03-26T02:59:41.807 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:41 smithi172 systemd[1]: Stopping Ceph osd.7 for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:42.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:41.964+0000 7fe9a3357700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:59:42.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:41.964+0000 7fe9a3357700 -1 osd.7 159 *** Got signal Terminated *** 2024-03-26T02:59:42.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:41 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:41.964+0000 7fe9a3357700 -1 osd.7 159 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:59:42.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:42.008+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:42.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:42.008+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:42.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:42.008+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:42.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:42 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:42.008+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:43.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:43.021+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:43.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:43.021+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:43.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:43.021+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:43.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:43.021+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:44.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:43.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=8 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi164.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:59:44.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:43.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi164.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.164:8443: connect: connection refused" 2024-03-26T02:59:44.044 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:43 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:43.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi172.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.172:8443: connect: connection refused" 2024-03-26T02:59:44.306 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:43.984+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:44.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:43.984+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:44.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:43.984+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:44.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:43 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:43.984+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:45.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:45.020+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:45.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:45.020+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:45.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:45.020+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:45.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:45.020+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:45.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:45 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:45.020+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6804 osd.4 since back 2024-03-26T02:59:22.803947+0000 front 2024-03-26T02:59:22.803967+0000 (oldest deadline 2024-03-26T02:59:44.503866+0000) 2024-03-26T02:59:46.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:46.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6806 osd.0 since back 2024-03-26T02:58:48.999675+0000 front 2024-03-26T02:58:48.999632+0000 (oldest deadline 2024-03-26T02:59:12.499488+0000) 2024-03-26T02:59:46.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:46.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6814 osd.1 since back 2024-03-26T02:58:57.700920+0000 front 2024-03-26T02:58:57.700854+0000 (oldest deadline 2024-03-26T02:59:20.600794+0000) 2024-03-26T02:59:46.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:46.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6822 osd.2 since back 2024-03-26T02:59:04.101749+0000 front 2024-03-26T02:59:04.101793+0000 (oldest deadline 2024-03-26T02:59:31.102020+0000) 2024-03-26T02:59:46.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:46.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.164:6830 osd.3 since back 2024-03-26T02:59:14.002934+0000 front 2024-03-26T02:59:14.002915+0000 (oldest deadline 2024-03-26T02:59:38.102868+0000) 2024-03-26T02:59:46.307 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:46 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7[93428]: 2024-03-26T02:59:46.040+0000 7fe99f94b700 -1 osd.7 159 heartbeat_check: no reply from 172.21.15.172:6804 osd.4 since back 2024-03-26T02:59:22.803947+0000 front 2024-03-26T02:59:22.803967+0000 (oldest deadline 2024-03-26T02:59:44.503866+0000) 2024-03-26T02:59:47.734 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:47 smithi172 podman[102946]: 2024-03-26 02:59:47.483044105 +0000 UTC m=+5.626846728 container died 66123d33bbf266b1cac6d6fbed4910acf490794c35ffa8ece9ea726c03f7fc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T02:59:48.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:47 smithi172 podman[102946]: 2024-03-26 02:59:47.806658768 +0000 UTC m=+5.950461381 container cleanup 66123d33bbf266b1cac6d6fbed4910acf490794c35ffa8ece9ea726c03f7fc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:59:48.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:47 smithi172 bash[102946]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7 2024-03-26T02:59:48.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:47 smithi172 podman[102963]: 2024-03-26 02:59:47.966294034 +0000 UTC m=+0.479084590 container remove 66123d33bbf266b1cac6d6fbed4910acf490794c35ffa8ece9ea726c03f7fc34 (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T02:59:48.694 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:48 smithi172 podman[103059]: 2024-03-26 02:59:48.366931915 +0000 UTC m=+0.110719698 container create 43dcc680ae750056eedd3f4f33bf65e48d7bd79df8327135e19a229d741bb0dd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T02:59:48.695 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:48 smithi172 podman[103059]: 2024-03-26 02:59:48.274274189 +0000 UTC m=+0.018061977 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T02:59:48.695 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:48 smithi172 podman[103059]: 2024-03-26 02:59:48.611000055 +0000 UTC m=+0.354787838 container init 43dcc680ae750056eedd3f4f33bf65e48d7bd79df8327135e19a229d741bb0dd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T02:59:48.695 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:48 smithi172 podman[103059]: 2024-03-26 02:59:48.61460007 +0000 UTC m=+0.358387857 container start 43dcc680ae750056eedd3f4f33bf65e48d7bd79df8327135e19a229d741bb0dd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-26T02:59:48.695 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:48 smithi172 podman[103059]: 2024-03-26 02:59:48.694215249 +0000 UTC m=+0.438003051 container attach 43dcc680ae750056eedd3f4f33bf65e48d7bd79df8327135e19a229d741bb0dd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T02:59:49.057 INFO:journalctl@ceph.osd.7.smithi172.stdout:Mar 26 02:59:48 smithi172 podman[103059]: 2024-03-26 02:59:48.901040035 +0000 UTC m=+0.644827821 container died 43dcc680ae750056eedd3f4f33bf65e48d7bd79df8327135e19a229d741bb0dd (image=quay.ceph.io/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T02:59:49.479 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T02:59:49.521 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:59:49.522 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T02:59:49.522 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T02:59:49.522 DEBUG:teuthology.orchestra.run.smithi172:> sudo systemctl stop ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T02:59:49.807 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 systemd[1]: Stopping Ceph prometheus.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:50.150 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.842Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.843Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.843Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T02:59:50.151 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a[64367]: ts=2024-03-26T02:59:49.843Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T02:59:50.152 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:49 smithi172 podman[103164]: 2024-03-26 02:59:49.852976012 +0000 UTC m=+0.103220203 container died 6e31603f6658434213d279238b5b4940c6bb0c5a09c8e4e7bada10d57998ee1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:50.152 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:50 smithi172 podman[103164]: 2024-03-26 02:59:50.150378846 +0000 UTC m=+0.400623026 container cleanup 6e31603f6658434213d279238b5b4940c6bb0c5a09c8e4e7bada10d57998ee1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:50.152 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:50 smithi172 bash[103164]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:59:50.498 INFO:journalctl@ceph.prometheus.a.smithi172.stdout:Mar 26 02:59:50 smithi172 podman[103178]: 2024-03-26 02:59:50.376504951 +0000 UTC m=+0.525756000 container remove 6e31603f6658434213d279238b5b4940c6bb0c5a09c8e4e7bada10d57998ee1f (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:50.509 DEBUG:teuthology.orchestra.run.smithi172:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T02:59:50.546 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:59:50.546 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T02:59:50.546 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:59:52.295 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:52 smithi164 systemd[1]: Stopping Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:52.589 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:52 smithi164 podman[131488]: 2024-03-26 02:59:52.323322199 +0000 UTC m=+0.117706676 container died 467a30dd8758ea35788b6db7fc2d5b4942ee0245ae33ededc5c9c0df065e1a0c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:52.955 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:52 smithi164 podman[131488]: 2024-03-26 02:59:52.696303244 +0000 UTC m=+0.490687726 container cleanup 467a30dd8758ea35788b6db7fc2d5b4942ee0245ae33ededc5c9c0df065e1a0c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:52.955 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:52 smithi164 bash[131488]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T02:59:53.272 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:52 smithi164 podman[131502]: 2024-03-26 02:59:52.955522701 +0000 UTC m=+0.633191500 container remove 467a30dd8758ea35788b6db7fc2d5b4942ee0245ae33ededc5c9c0df065e1a0c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:53.273 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:52 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:59:53.273 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:53 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T02:59:53.273 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:53 smithi164 systemd[1]: Stopped Ceph node-exporter.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:59:53.273 INFO:journalctl@ceph.node-exporter.a.smithi164.stdout:Mar 26 02:59:53 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 3.238s CPU time. 2024-03-26T02:59:53.525 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:53 smithi164 systemd[1]: Stopping Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T02:59:53.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:53 smithi164 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a[78783]: ts=2024-03-26T02:59:53.525Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:59:53.794 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:53 smithi164 podman[131619]: 2024-03-26 02:59:53.566509484 +0000 UTC m=+0.144221034 container died 763628284e16f991a5be32b28bee39f60aba703190f0bd1eaa148cb07a010f2c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:54.168 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:53 smithi164 podman[131619]: 2024-03-26 02:59:53.925371896 +0000 UTC m=+0.503083453 container cleanup 763628284e16f991a5be32b28bee39f60aba703190f0bd1eaa148cb07a010f2c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:54.168 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:53 smithi164 bash[131619]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:59:54.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:54 smithi164 podman[131633]: 2024-03-26 02:59:54.168037335 +0000 UTC m=+0.601512280 container remove 763628284e16f991a5be32b28bee39f60aba703190f0bd1eaa148cb07a010f2c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:59:54.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:54 smithi164 podman[131633]: 2024-03-26 02:59:54.243749622 +0000 UTC m=+0.677224567 volume remove e1275d27c34529ba0eb2d0d6bb9f665ac538e449dd8f1c388ad96b06a09e60ee 2024-03-26T02:59:54.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:54 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:59:54.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:54 smithi164 systemd[1]: Stopped Ceph alertmanager.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T02:59:54.544 INFO:journalctl@ceph.alertmanager.a.smithi164.stdout:Mar 26 02:59:54 smithi164 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.600s CPU time. 2024-03-26T03:00:17.357 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T03:00:19.062 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:18 smithi172 systemd[1]: Stopping Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T03:00:19.368 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:19 smithi172 podman[103573]: 2024-03-26 03:00:19.062042771 +0000 UTC m=+0.115264269 container died 352ab9d69d2a8cfa784a3d8a3f7b136ca37c227d13f7dddb82317eb352db561c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:00:19.368 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:19 smithi172 podman[103573]: 2024-03-26 03:00:19.368058376 +0000 UTC m=+0.421279880 container cleanup 352ab9d69d2a8cfa784a3d8a3f7b136ca37c227d13f7dddb82317eb352db561c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:00:19.721 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:19 smithi172 bash[103573]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T03:00:19.721 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:19 smithi172 podman[103587]: 2024-03-26 03:00:19.519209659 +0000 UTC m=+0.461023116 container remove 352ab9d69d2a8cfa784a3d8a3f7b136ca37c227d13f7dddb82317eb352db561c (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:00:19.721 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:19 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T03:00:20.057 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:19 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T03:00:20.057 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:19 smithi172 systemd[1]: Stopped Ceph node-exporter.b for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T03:00:20.057 INFO:journalctl@ceph.node-exporter.b.smithi172.stdout:Mar 26 03:00:19 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 3.179s CPU time. 2024-03-26T03:00:20.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:19 smithi172 systemd[1]: Stopping Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715... 2024-03-26T03:00:20.422 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=server t=2024-03-26T03:00:20.16353586Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-26T03:00:20.422 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=tracing t=2024-03-26T03:00:20.163939135Z level=info msg="Closing tracing" 2024-03-26T03:00:20.423 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a[66331]: logger=ticker t=2024-03-26T03:00:20.163943958Z level=info msg=stopped last_tick=2024-03-26T03:00:20Z 2024-03-26T03:00:20.423 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 podman[103694]: 2024-03-26 03:00:20.173868156 +0000 UTC m=+0.116241619 container died c46cfc9279a0762ac3525fb1317c7572bffc5019261651f6fc439e09b9537971 (image=quay.io/ceph/grafana:9.4.12, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:00:20.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 podman[103694]: 2024-03-26 03:00:20.513964365 +0000 UTC m=+0.456337837 container cleanup c46cfc9279a0762ac3525fb1317c7572bffc5019261651f6fc439e09b9537971 (image=quay.io/ceph/grafana:9.4.12, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:00:20.706 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 bash[103694]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a 2024-03-26T03:00:21.056 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 podman[103708]: 2024-03-26 03:00:20.70640894 +0000 UTC m=+0.535054931 container remove c46cfc9279a0762ac3525fb1317c7572bffc5019261651f6fc439e09b9537971 (image=quay.io/ceph/grafana:9.4.12, name=ceph-e2654d3c-eb16-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:00:21.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-26T03:00:21.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 systemd[1]: Stopped Ceph grafana.a for e2654d3c-eb16-11ee-95d0-87774f69a715. 2024-03-26T03:00:21.057 INFO:journalctl@ceph.grafana.a.smithi172.stdout:Mar 26 03:00:20 smithi172 systemd[1]: ceph-e2654d3c-eb16-11ee-95d0-87774f69a715@grafana.a.service: Consumed 12.081s CPU time. 2024-03-26T03:00:32.718 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:00:32.754 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:00:32.783 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T03:00:32.785 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/remote/smithi164/crash 2024-03-26T03:00:32.786 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/crash -- . 2024-03-26T03:00:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:tar: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T03:00:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:tar: Error is not recoverable: exiting now 2024-03-26T03:00:32.841 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/remote/smithi172/crash 2024-03-26T03:00:32.842 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/crash -- . 2024-03-26T03:00:32.877 INFO:teuthology.orchestra.run.smithi172.stderr:tar: /var/lib/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T03:00:32.877 INFO:teuthology.orchestra.run.smithi172.stderr:tar: Error is not recoverable: exiting now 2024-03-26T03:00:32.879 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T03:00:32.879 DEBUG:teuthology.orchestra.run.smithi164:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v OSD_DOWN | head -n 1 2024-03-26T03:00:32.912 INFO:teuthology.orchestra.run.smithi164.stdout:2024-03-26T02:31:54.373419+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T03:00:32.912 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T03:00:32.913 DEBUG:teuthology.orchestra.run.smithi164:> sudo egrep '\[SEC\]' /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v OSD_DOWN | head -n 1 2024-03-26T03:00:32.983 DEBUG:teuthology.orchestra.run.smithi164:> sudo egrep '\[ERR\]' /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v OSD_DOWN | head -n 1 2024-03-26T03:00:33.070 DEBUG:teuthology.orchestra.run.smithi164:> sudo egrep '\[WRN\]' /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v OSD_DOWN | head -n 1 2024-03-26T03:00:33.100 INFO:teuthology.orchestra.run.smithi164.stdout:2024-03-26T02:31:54.373419+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T03:00:33.101 INFO:tasks.cephadm:Compressing logs... 2024-03-26T03:00:33.101 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T03:00:33.142 DEBUG:teuthology.orchestra.run.smithi172:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T03:00:33.167 INFO:teuthology.orchestra.run.smithi164.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T03:00:33.167 INFO:teuthology.orchestra.run.smithi164.stderr:: No such file or directory 2024-03-26T03:00:33.167 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T03:00:33.167 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T03:00:33.168 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T03:00:33.168 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T03:00:33.170 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi164.cdxqhg.log 2024-03-26T03:00:33.170 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T03:00:33.170 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi164.cdxqhg.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T03:00:33.170 INFO:teuthology.orchestra.run.smithi164.stderr: 76.1% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi164.cdxqhg.log.gz 2024-03-26T03:00:33.170 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T03:00:33.170 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/tcmu-runner.log: 88.7% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T03:00:33.170 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T03:00:33.172 INFO:teuthology.orchestra.run.smithi172.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T03:00:33.172 INFO:teuthology.orchestra.run.smithi172.stderr:: No such file or directory 2024-03-26T03:00:33.172 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T03:00:33.172 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi172.gbbkmx.log 2024-03-26T03:00:33.172 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi172.gbbkmx.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr: 76.3% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-client.rgw.foo.smithi172.gbbkmx.log.gz 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mgr.x.log: /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.cephadm.log: 87.1% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log 2024-03-26T03:00:33.176 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T03:00:33.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log: 91.1% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T03:00:33.183 INFO:teuthology.orchestra.run.smithi172.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T03:00:33.183 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-volume.log: 89.1% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T03:00:33.185 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.0.log: 94.5% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T03:00:33.185 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log 2024-03-26T03:00:33.185 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.cephadm.log: 92.6% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T03:00:33.185 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T03:00:33.185 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T03:00:33.190 INFO:teuthology.orchestra.run.smithi164.stderr: 93.9% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T03:00:33.190 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T03:00:33.190 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T03:00:33.198 INFO:teuthology.orchestra.run.smithi172.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.b.log: 91.2% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T03:00:33.203 INFO:teuthology.orchestra.run.smithi172.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T03:00:33.224 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.1% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T03:00:33.330 INFO:teuthology.orchestra.run.smithi172.stderr: 89.8% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T03:00:33.674 INFO:teuthology.orchestra.run.smithi164.stderr: 89.1% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T03:00:33.929 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T03:00:33.993 INFO:teuthology.orchestra.run.smithi172.stderr: 92.5% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T03:00:34.932 INFO:teuthology.orchestra.run.smithi172.stderr: 93.2% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T03:00:34.984 INFO:teuthology.orchestra.run.smithi164.stderr: 93.3% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T03:00:35.044 INFO:teuthology.orchestra.run.smithi164.stderr: 91.0% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T03:00:35.068 INFO:teuthology.orchestra.run.smithi172.stderr: 93.8% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T03:00:35.079 INFO:teuthology.orchestra.run.smithi172.stderr: 93.4% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T03:00:35.345 INFO:teuthology.orchestra.run.smithi172.stderr: 93.5% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T03:00:35.347 INFO:teuthology.orchestra.run.smithi172.stderr: 2024-03-26T03:00:35.347 INFO:teuthology.orchestra.run.smithi172.stderr:real 0m2.190s 2024-03-26T03:00:35.347 INFO:teuthology.orchestra.run.smithi172.stderr:user 0m8.473s 2024-03-26T03:00:35.347 INFO:teuthology.orchestra.run.smithi172.stderr:sys 0m0.317s 2024-03-26T03:00:35.514 INFO:teuthology.orchestra.run.smithi164.stderr: 93.5% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T03:00:35.555 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T03:00:35.760 INFO:teuthology.orchestra.run.smithi164.stderr: 93.4% -- replaced with /var/log/ceph/e2654d3c-eb16-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T03:00:35.762 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-03-26T03:00:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m2.608s 2024-03-26T03:00:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m11.856s 2024-03-26T03:00:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.412s 2024-03-26T03:00:35.763 INFO:tasks.cephadm:Archiving logs... 2024-03-26T03:00:35.763 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/remote/smithi164/log 2024-03-26T03:00:35.764 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T03:00:37.387 DEBUG:teuthology.misc:Transferring archived files from smithi172:/var/log/ceph to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/remote/smithi172/log 2024-03-26T03:00:37.388 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T03:00:38.640 INFO:tasks.cephadm:Removing cluster... 2024-03-26T03:00:38.640 DEBUG:teuthology.orchestra.run.smithi164:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 --force 2024-03-26T03:00:38.933 DEBUG:teuthology.orchestra.run.smithi172:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2654d3c-eb16-11ee-95d0-87774f69a715 --force 2024-03-26T03:00:39.223 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T03:00:39.223 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T03:00:39.239 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T03:00:39.254 INFO:tasks.cephadm:Teardown complete 2024-03-26T03:00:39.254 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T03:00:39.265 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T03:00:39.266 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T03:00:39.281 DEBUG:teuthology.orchestra.run.smithi172:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T03:00:39.293 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-03-26T03:00:39.297 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T03:00:39.298 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-03-26T03:00:39.298 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 -388us[ -388us] +/- 84ms 2024-03-26T03:00:39.298 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv02.front.sepia.ceph.com 3 6 377 45 -478us[ -477us] +/- 52ms 2024-03-26T03:00:39.298 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +963us[ +963us] +/- 70ms 2024-03-26T03:00:39.298 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:00:39.307 INFO:teuthology.orchestra.run.smithi172.stderr:bash: line 1: ntpq: command not found 2024-03-26T03:00:39.310 INFO:teuthology.orchestra.run.smithi172.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T03:00:39.311 INFO:teuthology.orchestra.run.smithi172.stdout:=============================================================================== 2024-03-26T03:00:39.311 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 -402us[ -402us] +/- 84ms 2024-03-26T03:00:39.311 INFO:teuthology.orchestra.run.smithi172.stdout:^* hv02.front.sepia.ceph.com 3 6 377 44 -492us[ -489us] +/- 52ms 2024-03-26T03:00:39.311 INFO:teuthology.orchestra.run.smithi172.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +959us[ +959us] +/- 70ms 2024-03-26T03:00:39.311 INFO:teuthology.orchestra.run.smithi172.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:00:39.312 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T03:00:39.321 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T03:00:39.322 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T03:00:39.332 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T03:00:39.430 DEBUG:teuthology.orchestra.run.smithi172:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T03:00:39.510 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-26T03:00:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:39.542 DEBUG:teuthology.orchestra.run.smithi172:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-26T03:00:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:39.573 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T03:00:39.582 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T03:00:39.590 INFO:teuthology.task.internal:Duration was 2905.991529 seconds 2024-03-26T03:00:39.591 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T03:00:39.600 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T03:00:39.600 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T03:00:39.603 DEBUG:teuthology.orchestra.run.smithi172:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T03:00:39.643 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T03:00:39.655 INFO:teuthology.orchestra.run.smithi172.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T03:00:40.086 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T03:00:40.086 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-03-26T03:00:40.086 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T03:00:40.146 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi172.front.sepia.ceph.com 2024-03-26T03:00:40.146 DEBUG:teuthology.orchestra.run.smithi172:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T03:00:40.178 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T03:00:40.178 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T03:00:40.188 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T03:00:40.246 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T03:00:40.246 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T03:00:40.257 DEBUG:teuthology.orchestra.run.smithi172:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T03:00:40.840 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T03:00:40.851 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T03:00:40.852 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T03:00:40.878 DEBUG:teuthology.orchestra.run.smithi172:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T03:00:40.906 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T03:00:40.917 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T03:00:40.921 DEBUG:teuthology.orchestra.run.smithi172:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T03:00:40.944 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-03-26T03:00:40.972 INFO:teuthology.orchestra.run.smithi172.stdout:kernel.core_pattern = core 2024-03-26T03:00:41.002 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T03:00:41.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:41.020 DEBUG:teuthology.orchestra.run.smithi172:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T03:00:41.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:00:41.057 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T03:00:41.067 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T03:00:41.068 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/remote/smithi164 2024-03-26T03:00:41.068 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T03:00:41.125 DEBUG:teuthology.misc:Transferring archived files from smithi172:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368/remote/smithi172 2024-03-26T03:00:41.125 DEBUG:teuthology.orchestra.run.smithi172:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T03:00:41.168 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T03:00:41.169 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T03:00:41.171 DEBUG:teuthology.orchestra.run.smithi172:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T03:00:41.218 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T03:00:41.229 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T03:00:41.230 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T03:00:41.245 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T03:00:41.245 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T03:00:41.248 DEBUG:teuthology.orchestra.run.smithi172:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T03:00:41.261 INFO:teuthology.orchestra.run.smithi164.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 03:00 /home/ubuntu/cephtest 2024-03-26T03:00:41.272 INFO:teuthology.orchestra.run.smithi172.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 03:00 /home/ubuntu/cephtest 2024-03-26T03:00:41.273 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T03:00:41.284 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T03:00:41.310 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T03:00:41.341 DEBUG:teuthology.nuke:shortname: smithi164 2024-03-26T03:00:41.341 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T03:00:41.366 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 02:03:01.602044', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9Y+jJxG3SK/CYTEVLI5sHtamX0DnyUoB9Jg8lZWQEEnoq0QFqWuPnWjklx0lJurNmOgobtycmv3VaDE22/M9o='} 2024-03-26T03:00:41.370 DEBUG:teuthology.nuke:shortname: smithi172 2024-03-26T03:00:41.370 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T03:00:41.398 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi172.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622368', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 02:03:01.603484', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPopve8QBzZsUc6kyh4uSWmx3Pvp6NWnL9MgU1L9kbkSmqmqf4tyO4SXov0zAASIkbM+YdljgWvlfNwgq0AGbSc='} 2024-03-26T03:00:41.424 INFO:teuthology.orchestra.console.smithi164:Power off 2024-03-26T03:00:41.424 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T03:00:41.450 DEBUG:teuthology.orchestra.console.smithi164:power off output: Chassis Power Control: Down/Off 2024-03-26T03:00:41.450 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:00:41.465 INFO:teuthology.orchestra.console.smithi172:Power off 2024-03-26T03:00:41.466 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T03:00:41.475 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-26T03:00:41.488 DEBUG:teuthology.orchestra.console.smithi172:power off output: Chassis Power Control: Down/Off 2024-03-26T03:00:41.488 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:00:41.511 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-26T03:00:45.476 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:00:45.512 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:00:45.589 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-26T03:00:45.623 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-26T03:00:49.594 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:00:49.624 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:00:49.705 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is on 2024-03-26T03:00:49.737 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is on 2024-03-26T03:00:53.710 DEBUG:teuthology.orchestra.console.smithi164:pexpect command: ipmitool -H smithi164.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:00:53.738 DEBUG:teuthology.orchestra.console.smithi172:pexpect command: ipmitool -H smithi172.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T03:00:53.821 DEBUG:teuthology.orchestra.console.smithi164:check power output: Chassis Power is off 2024-03-26T03:00:53.849 DEBUG:teuthology.orchestra.console.smithi172:check power output: Chassis Power is off 2024-03-26T03:00:53.922 INFO:teuthology.orchestra.console.smithi164:Power off completed 2024-03-26T03:00:53.950 INFO:teuthology.orchestra.console.smithi172:Power off completed 2024-03-26T03:00:54.054 INFO:teuthology.run:Summary data: description: orch:cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/staggered 4-wait 5-upgrade-ls agent/off mon_election/classic} duration: 2905.9915285110474 failure_reason: '"2024-03-26T02:31:54.373419+0000 mon.a (mon.0) 968 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_adking@teuthology success: false 2024-03-26T03:00:54.054 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T03:00:54.133 INFO:teuthology.run:FAIL